diff --git a/.claude/rules/incusos-immutability.md b/.claude/rules/incusos-immutability.md new file mode 100644 index 0000000..428527b --- /dev/null +++ b/.claude/rules/incusos-immutability.md @@ -0,0 +1,88 @@ +--- +paths: + - "incusos/helpers/incusos-health" + - "notes/incusos-break-fix.md" +--- + +# IncusOS Immutability & Break-Fix Context + +## Architecture + +- IncusOS is an immutable OS with A/B partition scheme for safe updates. +- Version format: `YYYYMMDDHHMI` (build timestamp, e.g., `202602240349`). +- TPM-based full disk encryption: root + swap volumes unlocked via TPM + measured boot. Secure Boot enforced with PK, KEK, and 2x db certs. +- ZFS storage pool "local" on partition 11, contains volume "incus". +- Single SCSI disk per node (64 GiB in lab, QEMU HARDDISK). + +## Lab cluster nodes + +| Node | VMID | IP | Version | +|------|------|----|---------| +| node-01 | 900 | 192.168.102.140 | 202602240349 | +| node-02 | 901 | 192.168.102.141 | 202602230420 | +| node-03 | 902 | 192.168.102.142 | 202602230420 | + +- Management + cluster traffic on interface "mgmt" (static IPs, /22 subnet). +- DNS + gateway: 192.168.100.1. +- OVN enabled, all other services disabled. + +## IncusOS API endpoints + +All on port 8443, authenticated with Incus client certificate: + +- `/os/1.0` -- version, hostname, basic info +- `/os/1.0/system/security` -- TPM, Secure Boot, encryption/pool recovery keys +- `/os/1.0/system/storage` -- disks, partitions, ZFS pools +- `/os/1.0/system/resources` -- CPU, memory, hardware +- `/os/1.0/system/network` -- interfaces, DNS, routes +- `/os/1.0/system/update` -- channel, frequency, pending updates +- `/os/1.0/services/{ovn,iscsi,lvm,multipath,nvme,tailscale,usbip}` + +## incusos-health helper + +`incusos/helpers/incusos-health` queries the IncusOS API across all +cluster nodes. Actions: + +| Flag | Purpose | +|------|---------| +| `--status` | Version, hostname, TPM status, Secure Boot | +| `--partitions` | Disk/partition layout, A/B state | +| `--tpm` | TPM details, Secure Boot certs, encryption keys | +| `--services` | Enabled/disabled services | +| `--network` | Interface config, DNS, gateway | +| `--update` | Channel, frequency, pending updates | +| `--all` | All of the above | + +## Safety rules (MANDATORY) + +- **NEVER hard-stop during first boot** -- corrupts TPM permanently, + requires full reinstallation. Hard-stop is safe ONLY for already- + provisioned nodes (TPM state already sealed). +- **One node at a time** for break-fix exercises. 3-node cluster needs + 2/3 quorum. Taking down 2 nodes = cluster failure. +- **Proxmox snapshots** before every destructive test (VMID range 900-939). +- **Target node-03** (VMID 902) for destructive tests -- not the leader, + fewest workloads. +- **Verify health** with `incusos-health --all` before and after exercises. +- **Monitor via Grafana** during all exercises. + +## Update mechanics + +- `os_version` = current running version. +- `os_version_next` = version on standby partition (same as current if + no update pending, different if update downloaded). +- `needs_reboot` = true when update is downloaded and waiting for reboot. +- `auto_reboot: false` in lab -- updates download but don't reboot. +- Check frequency: 6 hours, stable channel. +- Nodes update independently (rolling, not coordinated). + +## Break-fix exercises (defined, not yet executed) + +1. **Normal Update Observation** -- watch download, reboot, A/B switch +2. **Simulated Failed Update** -- hard-stop node-03 mid-update, expect rollback +3. **Network Isolation** -- disconnect node-03 NIC, observe OVN tunnel loss +4. **Full Node Failure** -- hard-stop node-03, verify 2/3 quorum holds + +See `notes/incusos-break-fix.md` for full exercise details, prerequisites, +expected behavior, and recovery steps. diff --git a/CLAUDE.md b/CLAUDE.md index 29d2537..ea011a6 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -35,7 +35,8 @@ incus-contrib/ │ │ ├── proxmox-api # Authenticated API calls (handles ! in token) │ │ ├── aether-browser # Playwright browser automation for Aether web UI │ │ ├── ovn-inspect # OVN/OVS topology inspection (--nb/--sb/--ovs/--trace) -│ │ └── incus-mitm # API traffic capture and analysis (--capture/--live/--analyze) +│ │ ├── incus-mitm # API traffic capture and analysis (--capture/--live/--analyze) +│ │ └── incusos-health # IncusOS health inspection (--status/--tpm/--partitions/--all) │ ├── lab-test # Guided lab validation (12 test phases) │ ├── observe-deploy # Single-VM deploy with console screenshots │ ├── proxmox.yaml # Proxmox connection (gitignored) @@ -192,6 +193,7 @@ which files are being edited: | `observability.md` | deploy-observability, observability dashboards/guide | | `ovn-internals.md` | ovn-inspect, ovn-deep-dive | | `aether-api-payloads.md` | incus-mitm, api-interception | +| `incusos-immutability.md` | incusos-health, incusos-break-fix | | `proxmox-ssh-rules.md` | **Always loaded** (no paths filter) | | `lab-infrastructure.md` | incusos-proxmox, lab-test, examples | diff --git a/incusos/helpers/incusos-health b/incusos/helpers/incusos-health new file mode 100755 index 0000000..e0eca00 --- /dev/null +++ b/incusos/helpers/incusos-health @@ -0,0 +1,758 @@ +#!/usr/bin/env bash +# incusos-health -- Inspect IncusOS node health via the /os/1.0 REST API +# +# Queries one or more IncusOS cluster nodes for status, storage, security, +# services, network, and update information. +# +# Usage: incusos-health [OPTIONS] [ACTION] +# Run 'incusos-health --help' for full usage information. + +set -euo pipefail + +readonly VERSION="0.1.0" +readonly SCRIPT_NAME="incusos-health" + +# --------------------------------------------------------------------------- +# Defaults +# --------------------------------------------------------------------------- + +REMOTE="oc-node-01" +NODE="" +JSON_OUTPUT=false +DRY_RUN=false +VERBOSE=false +QUIET=false +ACTION="" + +# Services to check (IncusOS built-in services) +readonly SERVICES=(iscsi lvm multipath nvme ovn tailscale usbip) + +# --------------------------------------------------------------------------- +# Color and formatting +# --------------------------------------------------------------------------- + +setup_colors() { + if [[ -t 1 ]] && [[ "${NO_COLOR:-}" != "1" ]] && [[ "${TERM:-}" != "dumb" ]]; then + RED='\033[0;31m' + GREEN='\033[0;32m' + YELLOW='\033[0;33m' + BLUE='\033[0;34m' + CYAN='\033[0;36m' + BOLD='\033[1m' + DIM='\033[2m' + RESET='\033[0m' + else + RED='' GREEN='' YELLOW='' BLUE='' CYAN='' BOLD='' DIM='' RESET='' + fi +} + +info() { [[ "$QUIET" == true ]] && return; echo -e "${BLUE}[info]${RESET} $*"; } +success() { [[ "$QUIET" == true ]] && return; echo -e "${GREEN}[ok]${RESET} $*"; } +warn() { echo -e "${YELLOW}[warn]${RESET} $*" >&2; } +error() { echo -e "${RED}[error]${RESET} $*" >&2; } +step() { [[ "$QUIET" == true ]] && return; echo -e "${CYAN}[step]${RESET} ${BOLD}$*${RESET}"; } +detail() { [[ "$VERBOSE" != true ]] && return; echo -e "${DIM} $*${RESET}"; } + +# --------------------------------------------------------------------------- +# Help +# --------------------------------------------------------------------------- + +usage() { + cat <&2 + exit 1 + ;; + esac + shift + done + + if [[ -z "$ACTION" ]]; then + error "No action specified" + echo "Run '${SCRIPT_NAME} --help' for usage." >&2 + exit 1 + fi +} + +# --------------------------------------------------------------------------- +# Node discovery +# --------------------------------------------------------------------------- + +discover_nodes() { + if [[ -n "$NODE" ]]; then + echo "$NODE" + return + fi + + local members + members=$(incus cluster list "${REMOTE}:" -f csv 2>/dev/null \ + | cut -d',' -f1 \ + | sed 's/^ *//;s/ *$//') || true + + if [[ -z "$members" ]]; then + # Not a cluster or cannot list -- fall back to the remote itself + echo "$REMOTE" + return + fi + + echo "$members" +} + +# --------------------------------------------------------------------------- +# Query helper +# --------------------------------------------------------------------------- + +query_node() { + local node="$1" endpoint="$2" + + if [[ "$DRY_RUN" == true ]]; then + info "(dry-run) incus query ${node}:${endpoint}" + return 0 + fi + + detail "incus query ${node}:${endpoint}" + incus query "${node}:${endpoint}" 2>/dev/null || echo "{}" +} + +# --------------------------------------------------------------------------- +# Section: status +# --------------------------------------------------------------------------- + +show_status() { + local node="$1" + + local env_json resources_json security_json update_json + env_json=$(query_node "$node" "/os/1.0") + resources_json=$(query_node "$node" "/os/1.0/system/resources") + security_json=$(query_node "$node" "/os/1.0/system/security") + update_json=$(query_node "$node" "/os/1.0/system/update") + + if [[ "$DRY_RUN" == true ]]; then return; fi + + if [[ "$JSON_OUTPUT" == true ]]; then + python3 -c " +import json, sys +print(json.dumps({ + 'environment': json.loads(sys.argv[1]), + 'resources': json.loads(sys.argv[2]), + 'security': json.loads(sys.argv[3]), + 'update': json.loads(sys.argv[4]), +}, indent=2)) +" "$env_json" "$resources_json" "$security_json" "$update_json" + return + fi + + python3 -c " +import json, sys + +env = json.loads(sys.argv[1]).get('environment', json.loads(sys.argv[1])) +res = json.loads(sys.argv[2]) +sec_raw = json.loads(sys.argv[3]) +sec = sec_raw.get('state', sec_raw) +upd = json.loads(sys.argv[4]) + +hostname = env.get('hostname', 'unknown') +version = env.get('os_version', env.get('version', 'unknown')) +uptime_s = env.get('uptime', 0) + +# Format uptime +days = uptime_s // 86400 +hours = (uptime_s % 86400) // 3600 +mins = (uptime_s % 3600) // 60 +parts = [] +if days: parts.append(f'{days}d') +if hours: parts.append(f'{hours}h') +parts.append(f'{mins}m') +uptime_str = ' '.join(parts) + +# Memory +mem = res.get('memory', {}) +mem_used = mem.get('used', 0) +mem_total = mem.get('total', 0) + +def fmt_bytes(b): + for unit in ['B','KiB','MiB','GiB','TiB']: + if b < 1024: + return f'{b:.1f} {unit}' + b /= 1024 + return f'{b:.1f} PiB' + +mem_pct = (mem_used / mem_total * 100) if mem_total else 0 + +# Security (state level) +tpm_status = sec.get('tpm_status', 'unknown') +secboot_status = 'enabled' if sec.get('secure_boot_enabled', False) else 'disabled' +trusted = sec.get('system_state_is_trusted', 'unknown') + +# Update +needs_reboot = upd.get('state', {}).get('needs_reboot', False) +last_check = upd.get('state', {}).get('last_check', 'never') + +print(f' Hostname: {hostname}') +print(f' Version: {version}') +print(f' Uptime: {uptime_str}') +print(f' Memory: {fmt_bytes(mem_used)} / {fmt_bytes(mem_total)} ({mem_pct:.0f}%)') +print(f' TPM: {tpm_status}') +print(f' Secure Boot: {secboot_status}') +print(f' System trusted: {trusted}') +print(f' Needs reboot: {\"yes\" if needs_reboot else \"no\"}') +print(f' Last update: {last_check}') +" "$env_json" "$resources_json" "$security_json" "$update_json" +} + +# --------------------------------------------------------------------------- +# Section: partitions +# --------------------------------------------------------------------------- + +show_partitions() { + local node="$1" + + local storage_json security_json + storage_json=$(query_node "$node" "/os/1.0/system/storage") + security_json=$(query_node "$node" "/os/1.0/system/security") + + if [[ "$DRY_RUN" == true ]]; then return; fi + + if [[ "$JSON_OUTPUT" == true ]]; then + python3 -c " +import json, sys +print(json.dumps({ + 'storage': json.loads(sys.argv[1]), + 'security': json.loads(sys.argv[2]), +}, indent=2)) +" "$storage_json" "$security_json" + return + fi + + python3 -c " +import json, sys + +stor_raw = json.loads(sys.argv[1]) +sec_raw = json.loads(sys.argv[2]) +stor = stor_raw.get('state', stor_raw) +sec = sec_raw.get('state', sec_raw) + +def fmt_bytes(b): + for unit in ['B','KiB','MiB','GiB','TiB']: + if b < 1024: + return f'{b:.1f} {unit}' + b /= 1024 + return f'{b:.1f} PiB' + +# Drives +disks = stor.get('drives', stor.get('disks', [])) +if disks: + print(' Drives:') + for d in disks: + model = d.get('model_name', d.get('model', 'unknown')) + bus = d.get('bus', 'unknown') + size = d.get('capacity_in_bytes', d.get('size', 0)) + boot = ' [boot]' if d.get('boot') else '' + did = d.get('id', '?').split('/')[-1] + pool = d.get('member_pool', '') + pool_str = f' pool={pool}' if pool else '' + print(f' {did}: {model} ({bus}, {fmt_bytes(size)}){boot}{pool_str}') +else: + print(' Drives: none found') + +# ZFS pools +pools = stor.get('pools', []) +if pools: + print(' ZFS pools:') + for p in pools: + name = p.get('name', 'unknown') + state = p.get('state', 'unknown') + ptype = p.get('type', 'unknown') + alloc = p.get('pool_allocated_space_in_bytes', 0) + total = p.get('raw_pool_size_in_bytes', p.get('usable_pool_size_in_bytes', 0)) + pct = (alloc / total * 100) if total else 0 + enc_key = p.get('encryption_key_status', 'N/A') + print(f' {name}: {state} ({ptype}, {fmt_bytes(alloc)} / {fmt_bytes(total)}, {pct:.0f}%) enc_key={enc_key}') + for v in p.get('volumes', []): + vname = v.get('name', '?') + vuse = v.get('use', '?') + vusage = v.get('usage_in_bytes', 0) + vquota = v.get('quota_in_bytes', 0) + quota_str = fmt_bytes(vquota) if vquota else 'no quota' + print(f' vol {vname} ({vuse}): {fmt_bytes(vusage)} / {quota_str}') +else: + print(' ZFS pools: none found') + +# Encryption +encrypted_vols = sec.get('encrypted_volumes', []) +if encrypted_vols: + print(' Encryption:') + for v in encrypted_vols: + vol = v.get('volume', '?') + state = v.get('state', '?') + print(f' {vol}: {state}') +else: + print(' Encryption: not configured') +" "$storage_json" "$security_json" +} + +# --------------------------------------------------------------------------- +# Section: tpm +# --------------------------------------------------------------------------- + +show_tpm() { + local node="$1" + + local security_json + security_json=$(query_node "$node" "/os/1.0/system/security") + + if [[ "$DRY_RUN" == true ]]; then return; fi + + if [[ "$JSON_OUTPUT" == true ]]; then + python3 -c " +import json, sys +print(json.dumps(json.loads(sys.argv[1]), indent=2)) +" "$security_json" + return + fi + + python3 -c " +import json, sys + +sec_raw = json.loads(sys.argv[1]) +sec = sec_raw.get('state', sec_raw) +sec_cfg = sec_raw.get('config', {}) + +# TPM +tpm_status = sec.get('tpm_status', 'unknown') +print(f' TPM status: {tpm_status}') + +# Secure Boot +sb_enabled = sec.get('secure_boot_enabled', False) +print(f' Secure Boot: {\"enabled\" if sb_enabled else \"disabled\"}') + +# Certificates +certs = sec.get('secure_boot_certificates', []) +if certs: + print(' Certificates:') + for c in certs: + ctype = c.get('type', '?') + subject = c.get('subject', 'unknown') + fp = c.get('fingerprint', '') + fp_short = fp[:16] + '...' if len(fp) > 16 else fp + print(f' {ctype:4s} {subject}') + if fp_short: + print(f' fingerprint: {fp_short}') + +# Encrypted volumes +enc_vols = sec.get('encrypted_volumes', []) +if enc_vols: + print(' Encrypted volumes:') + for v in enc_vols: + print(f' {v.get(\"volume\", \"?\")}: {v.get(\"state\", \"?\")}') + +# Recovery keys +recovery_keys = sec_cfg.get('encryption_recovery_keys', []) +retrieved = sec.get('encryption_recovery_keys_retrieved', False) +print(f' Recovery keys: {len(recovery_keys)} key(s), retrieved={retrieved}') + +# Pool recovery keys +pool_keys = sec.get('pool_recovery_keys', {}) +if pool_keys: + print(f' Pool keys: {list(pool_keys.keys())}') + +# System trusted +trusted = sec.get('system_state_is_trusted', 'unknown') +print(f' System trusted: {trusted}') +" "$security_json" +} + +# --------------------------------------------------------------------------- +# Section: services +# --------------------------------------------------------------------------- + +show_services() { + local node="$1" + + if [[ "$JSON_OUTPUT" == true ]]; then + local all_json="{" + local first=true + for svc in "${SERVICES[@]}"; do + local svc_json + svc_json=$(query_node "$node" "/os/1.0/services/${svc}") + if [[ "$DRY_RUN" == true ]]; then continue; fi + if [[ "$first" == true ]]; then + first=false + else + all_json+="," + fi + all_json+="\"${svc}\":${svc_json}" + done + all_json+="}" + if [[ "$DRY_RUN" != true ]]; then + echo "$all_json" | python3 -m json.tool + fi + return + fi + + for svc in "${SERVICES[@]}"; do + local svc_json + svc_json=$(query_node "$node" "/os/1.0/services/${svc}") + if [[ "$DRY_RUN" == true ]]; then continue; fi + + python3 -c " +import json, sys + +svc_name = sys.argv[1] +data = json.loads(sys.argv[2]) + +config = data.get('config', data) +enabled = config.get('enabled', False) +status = 'enabled' if enabled else 'disabled' + +line = f' {svc_name:12s} {status}' +print(line) + +# Extra details for OVN +if svc_name == 'ovn' and enabled: + db = config.get('database', '') + tunnel_addr = config.get('tunnel_address', '') + tunnel_proto = config.get('tunnel_protocol', '') + if db: + print(f' database: {db}') + if tunnel_addr: + print(f' tunnel address: {tunnel_addr}') + if tunnel_proto: + print(f' tunnel protocol: {tunnel_proto}') +" "$svc" "$svc_json" + done +} + +# --------------------------------------------------------------------------- +# Section: network +# --------------------------------------------------------------------------- + +show_network() { + local node="$1" + + local network_json + network_json=$(query_node "$node" "/os/1.0/system/network") + + if [[ "$DRY_RUN" == true ]]; then return; fi + + if [[ "$JSON_OUTPUT" == true ]]; then + echo "$network_json" | python3 -m json.tool + return + fi + + python3 -c " +import json, sys + +net = json.loads(sys.argv[1]) + +def fmt_bytes(b): + for unit in ['B','KiB','MiB','GiB','TiB']: + if b < 1024: return f'{b:.1f} {unit}' + b /= 1024 + return f'{b:.1f} PiB' + +# Interfaces +ifaces = net.get('interfaces', net.get('config', {}).get('interfaces', [])) +if not isinstance(ifaces, list): ifaces = [] +print(' Interfaces:') +for iface in ifaces: + name = iface.get('name', 'unknown') + hwaddr = iface.get('hwaddr', iface.get('mac', '')) + roles = iface.get('roles', []) + roles_str = ', '.join(roles) if isinstance(roles, list) and roles else '' + addrs = iface.get('addresses', []) + addr_strs = [a if isinstance(a, str) else a.get('address', '?') for a in addrs] + state = iface.get('state', {}) + + header = f' {name}' + if hwaddr: header += f' ({hwaddr})' + if roles_str: header += f' [{roles_str}]' + print(header) + + if addr_strs: + print(f' addresses: {\", \".join(addr_strs)}') + + link_state = state.get('state', state.get('link', '')) + mtu = state.get('mtu', iface.get('mtu', '')) + speed = state.get('speed', '') + if link_state: + parts = [f'state={link_state}'] + if mtu: parts.append(f'mtu={mtu}') + if speed: parts.append(f'speed={speed}') + print(f' link: {\", \".join(parts)}') + + routes = iface.get('routes', []) + if isinstance(routes, list) and routes: + route_strs = [] + for r in routes: + if isinstance(r, str): route_strs.append(r) + elif isinstance(r, dict): + s = r.get('destination', r.get('to', '?')) + via = r.get('via', r.get('gateway', '')) + if via: s += f' via {via}' + route_strs.append(s) + if route_strs: + print(f' routes: {route_strs[0]}') + for rs in route_strs[1:]: + print(f' {rs}') + + stats = state.get('stats', iface.get('stats', {})) + if isinstance(stats, dict) and stats: + rx = stats.get('rx_bytes', stats.get('rx', 0)) + tx = stats.get('tx_bytes', stats.get('tx', 0)) + if rx or tx: + print(f' stats: rx={fmt_bytes(rx)}, tx={fmt_bytes(tx)}') + +# DNS +dns = net.get('dns', net.get('config', {}).get('dns', {})) +if isinstance(dns, dict) and dns: + print(' DNS:') + hostname = dns.get('hostname', dns.get('search', '')) + nameservers = dns.get('nameservers', dns.get('servers', [])) + if hostname: print(f' hostname: {hostname}') + if isinstance(nameservers, list) and nameservers: + print(f' nameservers: {\", \".join(str(n) for n in nameservers)}') +" "$network_json" +} + +# --------------------------------------------------------------------------- +# Section: update +# --------------------------------------------------------------------------- + +show_update() { + local node="$1" + + local update_json env_json + update_json=$(query_node "$node" "/os/1.0/system/update") + env_json=$(query_node "$node" "/os/1.0") + + if [[ "$DRY_RUN" == true ]]; then return; fi + + if [[ "$JSON_OUTPUT" == true ]]; then + echo "$update_json" | python3 -m json.tool + return + fi + + python3 -c " +import json, sys + +upd = json.loads(sys.argv[1]) +env = json.loads(sys.argv[2]).get('environment', json.loads(sys.argv[2])) +config = upd.get('config', {}) +state = upd.get('state', {}) + +channel = config.get('channel', 'unknown') +frequency = config.get('check_frequency', 'unknown') +auto_reboot = config.get('auto_reboot', 'unknown') + +current = env.get('os_version', 'unknown') +next_ver = env.get('os_version_next', 'none') +update_status = state.get('status', '') +last_check = state.get('last_check', 'never') +needs_reboot = state.get('needs_reboot', False) + +print(f' Channel: {channel}') +print(f' Check freq: {frequency}') +print(f' Auto reboot: {auto_reboot}') +print(f' Current: {current}') +print(f' Next version: {next_ver}') +print(f' Update status: {update_status}') +print(f' Last check: {last_check}') +print(f' Needs reboot: {\"yes\" if needs_reboot else \"no\"}') +" "$update_json" "$env_json" +} + +# --------------------------------------------------------------------------- +# Run a section across all nodes +# --------------------------------------------------------------------------- + +run_section() { + local section_name="$1" + local section_fn="$2" + shift 2 + local nodes=("$@") + + step "$section_name" + + for node in "${nodes[@]}"; do + echo -e " ${BOLD}--- ${node} ---${RESET}" + "$section_fn" "$node" + echo + done +} + +run_section_json() { + local section_fn="$1" + shift + local nodes=("$@") + + if [[ ${#nodes[@]} -eq 1 ]]; then + "$section_fn" "${nodes[0]}" + else + echo "{" + local first=true + for node in "${nodes[@]}"; do + if [[ "$first" == true ]]; then + first=false + else + echo "," + fi + echo " \"${node}\":" + "$section_fn" "$node" | sed 's/^/ /' + done + echo "}" + fi +} + +# --------------------------------------------------------------------------- +# Main +# --------------------------------------------------------------------------- + +main() { + setup_colors + parse_args "$@" + + # Discover cluster nodes + if [[ "$DRY_RUN" == true ]]; then + if [[ -n "$NODE" ]]; then + info "(dry-run) Would query node: ${NODE}" + else + info "(dry-run) Would discover nodes via: incus cluster list ${REMOTE}: -f csv" + fi + fi + + local nodes_raw + nodes_raw=$(discover_nodes) + + local nodes=() + while IFS= read -r line; do + [[ -n "$line" ]] && nodes+=("$line") + done <<< "$nodes_raw" + + if [[ ${#nodes[@]} -eq 0 ]]; then + error "No nodes found. Check that remote '${REMOTE}' is accessible." + error " incus remote list" + exit 1 + fi + + if [[ "$DRY_RUN" != true ]]; then + info "Nodes: ${nodes[*]}" + fi + echo + + # Section lookup: action -> (title, function) + local -A SECTION_TITLE=( + [status]="Status" [partitions]="Storage & Partitions" + [tpm]="TPM & Security" [services]="Services" + [network]="Network" [update]="Update" + ) + + # Build section list + local sections=() + if [[ "$ACTION" == "all" ]]; then + sections=(status partitions tpm services network update) + else + sections=("$ACTION") + fi + + # Dispatch + if [[ "$JSON_OUTPUT" == true && ${#sections[@]} -gt 1 ]]; then + echo "{" + local i=0 + for section in "${sections[@]}"; do + echo " \"${section}\":" + run_section_json "show_${section}" "${nodes[@]}" | sed 's/^/ /' + i=$((i + 1)) + [[ $i -lt ${#sections[@]} ]] && echo "," + done + echo "}" + else + for section in "${sections[@]}"; do + if [[ "$JSON_OUTPUT" == true ]]; then + run_section_json "show_${section}" "${nodes[@]}" + else + run_section "${SECTION_TITLE[$section]}" "show_${section}" "${nodes[@]}" + fi + done + fi +} + +main "$@" diff --git a/notes/incusos-break-fix.md b/notes/incusos-break-fix.md new file mode 100644 index 0000000..b2f3e9b --- /dev/null +++ b/notes/incusos-break-fix.md @@ -0,0 +1,421 @@ +# IncusOS Break-Fix Lab -- Immutability Exploration & Resilience Testing + +IncusOS is an immutable, purpose-built operating system for running Incus +clusters. This guide documents what we discovered about its internals through +the `/os/1.0` API, and defines a catalog of break-fix exercises for testing +cluster resilience in a safe lab environment. + +All observations come from a 3-node Proxmox-hosted cluster (VMID 900-902) +running IncusOS with virtual TPM, Secure Boot, and OVN networking. + +## IncusOS Architecture + +IncusOS is an immutable OS designed for a single purpose: running Incus. +Key architectural properties discovered via the `/os/1.0` API: + +- **Immutable root filesystem** with A/B partition scheme. The running system + boots from one partition; updates install to the other. Rollback is automatic + if the new partition fails validation. +- **TPM-based full disk encryption** -- root and swap volumes are encrypted + and unlocked automatically via TPM measured boot. No passphrase required + during normal operation. +- **Secure Boot enforced** with 4 certificates in the UEFI firmware: + - PK (Platform Key) -- root of trust + - KEK (Key Exchange Key) -- authorizes db updates + - db (2025 signing cert) -- validates current signed binaries + - db (2026 signing cert) -- validates next-year signed binaries +- **Version format**: `YYYYMMDDHHMI` (build timestamp, not semver). + Example: `202602240349` = 2026-02-24 at 03:49 UTC. +- **ZFS storage**: pool "local" on a dedicated partition (raid0 on single + disk, ~30 GiB usable), encrypted with its own pool recovery key. +- **Update system**: stable channel, 6-hour check frequency, + `auto_reboot: false` (updates download but do not reboot automatically). + +### Lab cluster state + +| Node | VMID | IP | IncusOS Version | +|------|------|----|-----------------| +| node-01 | 900 | 192.168.102.140 | 202602240349 | +| node-02 | 901 | 192.168.102.141 | 202602230420 | +| node-03 | 902 | 192.168.102.142 | 202602230420 | + +node-01 received a newer build than nodes 02/03, proving that nodes update +independently (rolling updates, not cluster-wide atomic upgrades). + +## Partition & Disk Layout + +Each node has a single 64 GiB SCSI disk (QEMU HARDDISK) with this layout: + +``` ++------------------------------------------------------+ +| SCSI Disk (64 GiB, QEMU HARDDISK) | +|------------------------------------------------------| +| Partition 1 EFI System Partition | +| Partition A Root filesystem (active or standby) | +| Partition B Root filesystem (standby or active) | +| Partition X Swap (encrypted, TPM-unlocked) | +| Partition 11 ZFS data pool ("local") | ++------------------------------------------------------+ +``` + +Key observations from the storage API: + +- **Partition 11** is the ZFS data partition, hosting pool "local". +- **Pool "local"**: raid0, single vdev, ~30 GiB total, contains volume + "incus" (~10 GiB used for Incus database, images, instances). +- **Root + swap**: both encrypted, both unlocked by TPM at boot. + No manual key entry needed unless TPM state is corrupted. + +## Security Chain + +The boot security chain, as validated through the API: + +``` +UEFI firmware + --> Secure Boot certificate validation (PK -> KEK -> db certs) + --> Signed kernel + initrd loaded + --> TPM measured boot (PCR measurements recorded) + --> TPM validates measurements match expected policy + --> Root partition decrypted and mounted + --> Swap partition decrypted and activated + --> System boots into trusted state +``` + +API-reported security state: + +| Property | Value | +|----------|-------| +| TPM status | ok | +| Secure Boot | enabled, enforced | +| System state | trusted | +| Encryption recovery keys | 1 key (retrievable via API) | +| Pool recovery keys | 1 key for pool "local" | + +**Recovery keys** are a safety net. If TPM state is corrupted (e.g., by a +hard-stop during first boot), the encryption recovery key allows manual +unlock. The pool recovery key allows ZFS pool import on a different system. + +## Services Configuration + +Discovered via `/os/1.0/services/*`: + +| Service | Status | Notes | +|---------|--------|-------| +| OVN | enabled | Connects to cluster DB at `tcp:192.168.102.142:6642`, Geneve tunnels | +| iSCSI | disabled | | +| LVM | disabled | | +| Multipath | disabled | | +| NVMe | disabled | | +| Tailscale | disabled | | +| USB/IP | disabled | | + +OVN is the only enabled service beyond Incus itself, providing the overlay +network for cross-node container connectivity and network policies. + +## Network Configuration + +Each node has a single management interface: + +| Property | Value | +|----------|-------| +| Interface name | mgmt | +| Role | management + cluster traffic | +| Addressing | Static | +| Subnet | /22 (192.168.100.0/22) | +| DNS | 192.168.100.1 | +| Gateway | 192.168.100.1 | + +Node IP assignments: +- node-01: `192.168.102.140/22` +- node-02: `192.168.102.141/22` +- node-03: `192.168.102.142/22` + +The management interface carries both management traffic (API, cluster +heartbeats) and OVN Geneve tunnel traffic. In production, these should +be separated. + +## Update Mechanics (Observed) + +The IncusOS update system uses A/B partitions for safe, rollback-capable +updates. Observations from querying the update API: + +1. **No pending update**: `os_version` and `os_version_next` are identical. + This means no update has been downloaded or is waiting for reboot. + +2. **Pending update**: `os_version_next` would differ from `os_version`, + indicating a new build has been downloaded to the standby partition. + +3. **Auto-reboot disabled**: `auto_reboot: false` means updates download + to the standby partition but the node continues running the current + version until explicitly rebooted (or until an admin triggers reboot + via API / UI). + +4. **No reboot needed**: `needs_reboot: false` confirms no downloaded + update is waiting for a reboot to activate. + +5. **Independent node updates**: node-01 is on `202602240349` while + nodes 02/03 are on `202602230420`. This proves each node checks for + and applies updates independently. There is no cluster-wide coordinated + update mechanism at the OS level. + +6. **Check frequency**: 6 hours. The stable channel is checked + automatically on this interval. + +### Update lifecycle (theoretical) + +``` +Check timer fires (every 6h) + --> Query stable channel for new version + --> Download new rootfs to standby partition (A or B) + --> os_version_next updated, needs_reboot = true + --> (if auto_reboot) Reboot automatically + --> (if !auto_reboot) Wait for manual reboot + --> On reboot: boot from new partition + --> TPM re-measures, validates new boot chain + --> If valid: new partition becomes active + --> If invalid: rollback to previous partition +``` + +## Break-Fix Exercise Catalog + +These exercises are designed to test IncusOS cluster resilience in the +Proxmox lab. All exercises follow strict safety rules (see Safety Rules +section below). + +**Current status**: All exercises are defined but not yet executed. + +--- + +### Exercise 1: Normal Update Observation + +**Goal**: Observe the full update lifecycle -- download, reboot, A/B +partition switch, version verification. + +**Prerequisites**: +- Proxmox snapshot of all cluster nodes (VMID 900-902) +- Grafana monitoring active (observe cluster metrics during update) +- Verify cluster health with `incusos-health --all` + +**Steps**: +1. Record current versions: `incusos-health --update` +2. Trigger update via Operations Center UI, or wait for 6h check interval +3. Monitor via Grafana for download activity and node state changes +4. When `needs_reboot: true`, reboot one node at a time +5. After reboot, verify new version: `incusos-health --status` +6. Confirm A/B partition switch: `incusos-health --partitions` + +**What to observe**: +- Download phase duration +- Reboot duration (typically 30-60s for IncusOS) +- Cluster behavior while one node reboots (2/3 quorum maintained) +- New version number in `os_version` +- Previous version still available on standby partition + +**Status**: Defined, not yet executed. + +--- + +### Exercise 2: Simulated Failed Update (Hard-Stop Mid-Update) + +**Goal**: Verify that IncusOS rolls back to the previous partition after +a failed update (simulated by hard-stopping the VM during update). + +**Prerequisites**: +- Proxmox snapshot of node-03 (VMID 902) -- non-leader, fewest workloads +- Verify node-03 has no critical workloads +- Confirm 3/3 nodes healthy before starting + +**Target node**: node-03 (VMID 902) ONLY. Never the cluster leader. + +**Steps**: +1. Take Proxmox snapshot of node-03: `incusos/helpers/proxmox-api POST /nodes/pve/qemu/902/snapshot -d '{"snapname":"pre-break-fix-2"}'` +2. Trigger an update on node-03 (if one is available) +3. During the update download or early reboot phase, hard-stop node-03 + via Proxmox API: `incusos/helpers/proxmox-api POST /nodes/pve/qemu/902/status/stop` +4. Wait 10 seconds, then start node-03: + `incusos/helpers/proxmox-api POST /nodes/pve/qemu/902/status/start` +5. Monitor boot via console screenshots: `incusos/helpers/proxmox-screenshot 902` +6. After boot, check version: `incusos-health --status` on node-03 + +**Expected behavior**: +- Node boots from the previous (known-good) partition +- Failed partition is marked as bad / not bootable +- `os_version` returns to the pre-update version +- Node rejoins the cluster automatically + +**Safety**: +- ONLY node-03 -- maintains 2/3 quorum (node-01 + node-02 continue) +- Do NOT hard-stop during *first boot* (corrupts TPM permanently) +- If recovery fails, restore from Proxmox snapshot + +**Status**: Defined, not yet executed. + +--- + +### Exercise 3: Network Isolation + +**Goal**: Observe how the cluster handles a node losing network +connectivity -- OVN tunnel loss, cluster membership changes, and +automatic recovery on reconnection. + +**Prerequisites**: +- Evacuate all workloads from node-03 before disconnecting +- Grafana monitoring active (watch OVN tunnel metrics, cluster events) +- Verify cluster health: `incusos-health --all` + +**Steps**: +1. Evacuate workloads from node-03: + `incus cluster evacuate oc-node-03 --target oc-node-01` +2. Disconnect node-03 NIC via Proxmox API: + `incusos/helpers/proxmox-api PUT /nodes/pve/qemu/902/config -d '{"net0":"virtio=...,link_down=1"}'` +3. Observe in Grafana: OVN Geneve tunnel loss, cluster detecting missing node +4. Wait 2-5 minutes for cluster to mark node-03 as offline +5. Reconnect NIC: set `link_down=0` via Proxmox API +6. Monitor recovery: node-03 should rejoin cluster, OVN tunnels re-establish +7. Measure total recovery time from reconnection to healthy state + +**Expected behavior**: +- Cluster detects node-03 offline within heartbeat timeout +- OVN tunnels from/to node-03 fail (Geneve encap packets lost) +- Cluster continues operating with 2/3 quorum +- On reconnection: node-03 rejoins, tunnels re-establish, workloads + can be restored + +**Recovery**: +- If node-03 does not rejoin: check OVN service status, restart if needed +- If cluster state is inconsistent: restore from Proxmox snapshot + +**Status**: Defined, not yet executed. + +--- + +### Exercise 4: Full Node Failure + +**Goal**: Verify that the cluster survives a complete node loss and +maintains operations with 2/3 quorum. Measure cluster rejoin time +after node recovery. + +**Prerequisites**: +- Proxmox snapshot of node-03 (VMID 902) +- Evacuate ALL workloads from node-03 +- Verify cluster health: `incusos-health --all` + +**Steps**: +1. Evacuate workloads: `incus cluster evacuate oc-node-03` +2. Take Proxmox snapshot of node-03 +3. Hard-stop node-03 via Proxmox: + `incusos/helpers/proxmox-api POST /nodes/pve/qemu/902/status/stop` +4. Verify cluster continues operating: + - `incus cluster list` should show 2 online, 1 offline + - Existing workloads on node-01/02 remain accessible + - OVN gateway should failover if node-03 was a gateway +5. Wait 5 minutes, observing Grafana metrics +6. Restart node-03: + `incusos/helpers/proxmox-api POST /nodes/pve/qemu/902/status/start` +7. Monitor boot: `incusos/helpers/proxmox-screenshot 902` +8. Measure time from start to cluster rejoin (node shows online) +9. Restore workloads: `incus cluster restore oc-node-03` + +**Expected behavior**: +- 2/3 quorum maintained -- all cluster operations continue +- OVN gateway failover occurs if node-03 was elected gateway +- After restart: node-03 boots, TPM unlocks, Incus starts, rejoins cluster +- Typical rejoin time: 1-3 minutes after boot + +**Safety**: +- This is NOT a first-boot scenario -- hard-stop is safe for already- + provisioned nodes (TPM state is already sealed) +- One node at a time ONLY +- If node-03 fails to rejoin after restart, restore snapshot + +**Status**: Defined, not yet executed. + +## Safety Rules + +These rules are non-negotiable for all break-fix exercises: + +1. **Never hard-stop during first boot.** First boot seals TPM measurements + and writes encryption keys. Interrupting this corrupts TPM state + permanently, requiring full reinstallation. + +2. **One node at a time.** A 3-node cluster requires 2/3 quorum. Never + take down more than one node simultaneously, or the cluster loses + quorum and all operations halt. + +3. **Proxmox snapshots before every destructive test.** Snapshot the + target node's VM (VMID 900-939) before any exercise that involves + stopping, disconnecting, or modifying the node. + +4. **Verify cluster health before and after.** Run `incusos-health --all` + (or equivalent API checks) before starting an exercise and after + completing it. Do not proceed if the cluster is already degraded. + +5. **Monitor via Grafana during all exercises.** Visual monitoring catches + issues that API polling might miss (e.g., OVN tunnel flapping, + storage I/O spikes). + +6. **Target node-03 for destructive tests.** node-03 (VMID 902) is the + preferred target because it is not the cluster leader and typically + has the fewest workloads. Evacuate before testing. + +7. **Keep recovery keys accessible.** The encryption recovery key and + ZFS pool recovery key should be retrieved and stored securely before + any exercise that might corrupt TPM state. + +## Helper Script: incusos-health + +The `incusos/helpers/incusos-health` script queries the IncusOS API on +cluster nodes to report system state. It is the primary tool for verifying +cluster health before and after break-fix exercises. + +### Usage + +```bash +incusos/helpers/incusos-health [ACTION] [OPTIONS] +``` + +### Actions + +| Action | Description | +|--------|-------------| +| `--status` | Basic system info: version, hostname, TPM status, Secure Boot | +| `--partitions` | Disk and partition layout, A/B partition state | +| `--tpm` | TPM details, Secure Boot certificates, encryption keys | +| `--services` | Enabled/disabled services (OVN, iSCSI, LVM, etc.) | +| `--network` | Network interface configuration, DNS, gateway | +| `--update` | Update channel, check frequency, pending update status | +| `--all` | Run all of the above in sequence | + +### Example output workflow + +```bash +# Before exercise: verify all nodes healthy +incusos/helpers/incusos-health --all + +# After exercise: verify recovery +incusos/helpers/incusos-health --status # Quick version check +incusos/helpers/incusos-health --all # Full health report +``` + +## API Reference + +All IncusOS system information is available via the REST API on each node. + +| Endpoint | Returns | +|----------|---------| +| `/os/1.0` | Version, hostname, basic system info | +| `/os/1.0/system/security` | TPM status, Secure Boot, encryption keys | +| `/os/1.0/system/storage` | Disks, partitions, ZFS pools | +| `/os/1.0/system/resources` | CPU, memory, hardware info | +| `/os/1.0/system/network` | Interfaces, DNS, routes | +| `/os/1.0/system/update` | Update channel, version, pending updates | +| `/os/1.0/services/ovn` | OVN configuration and status | +| `/os/1.0/services/iscsi` | iSCSI configuration | +| `/os/1.0/services/lvm` | LVM configuration | +| `/os/1.0/services/multipath` | Multipath configuration | +| `/os/1.0/services/nvme` | NVMe-oF configuration | +| `/os/1.0/services/tailscale` | Tailscale VPN configuration | +| `/os/1.0/services/usbip` | USB/IP configuration | + +The API listens on the management interface, HTTPS, port 8443 (same as +Incus). Authentication uses the Incus client certificate.