Add IncusOS break-fix guide, incusos-health helper, immutability rules
Phase 4: Explore IncusOS immutability via the /os/1.0 API: - incusos-health script with --status/--partitions/--tpm/--services/--network/--update - Validated: TPM encryption (root+swap), Secure Boot with 4 certs, ZFS pool - Documented A/B partition scheme, update mechanics, security chain - Defined 4 break-fix exercises (update observation, failed update, network isolation, full node failure) with safety rules and pickup points Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
cc017b648d
commit
95316a5bee
|
|
@ -0,0 +1,88 @@
|
|||
---
|
||||
paths:
|
||||
- "incusos/helpers/incusos-health"
|
||||
- "notes/incusos-break-fix.md"
|
||||
---
|
||||
|
||||
# IncusOS Immutability & Break-Fix Context
|
||||
|
||||
## Architecture
|
||||
|
||||
- IncusOS is an immutable OS with A/B partition scheme for safe updates.
|
||||
- Version format: `YYYYMMDDHHMI` (build timestamp, e.g., `202602240349`).
|
||||
- TPM-based full disk encryption: root + swap volumes unlocked via TPM
|
||||
measured boot. Secure Boot enforced with PK, KEK, and 2x db certs.
|
||||
- ZFS storage pool "local" on partition 11, contains volume "incus".
|
||||
- Single SCSI disk per node (64 GiB in lab, QEMU HARDDISK).
|
||||
|
||||
## Lab cluster nodes
|
||||
|
||||
| Node | VMID | IP | Version |
|
||||
|------|------|----|---------|
|
||||
| node-01 | 900 | 192.168.102.140 | 202602240349 |
|
||||
| node-02 | 901 | 192.168.102.141 | 202602230420 |
|
||||
| node-03 | 902 | 192.168.102.142 | 202602230420 |
|
||||
|
||||
- Management + cluster traffic on interface "mgmt" (static IPs, /22 subnet).
|
||||
- DNS + gateway: 192.168.100.1.
|
||||
- OVN enabled, all other services disabled.
|
||||
|
||||
## IncusOS API endpoints
|
||||
|
||||
All on port 8443, authenticated with Incus client certificate:
|
||||
|
||||
- `/os/1.0` -- version, hostname, basic info
|
||||
- `/os/1.0/system/security` -- TPM, Secure Boot, encryption/pool recovery keys
|
||||
- `/os/1.0/system/storage` -- disks, partitions, ZFS pools
|
||||
- `/os/1.0/system/resources` -- CPU, memory, hardware
|
||||
- `/os/1.0/system/network` -- interfaces, DNS, routes
|
||||
- `/os/1.0/system/update` -- channel, frequency, pending updates
|
||||
- `/os/1.0/services/{ovn,iscsi,lvm,multipath,nvme,tailscale,usbip}`
|
||||
|
||||
## incusos-health helper
|
||||
|
||||
`incusos/helpers/incusos-health` queries the IncusOS API across all
|
||||
cluster nodes. Actions:
|
||||
|
||||
| Flag | Purpose |
|
||||
|------|---------|
|
||||
| `--status` | Version, hostname, TPM status, Secure Boot |
|
||||
| `--partitions` | Disk/partition layout, A/B state |
|
||||
| `--tpm` | TPM details, Secure Boot certs, encryption keys |
|
||||
| `--services` | Enabled/disabled services |
|
||||
| `--network` | Interface config, DNS, gateway |
|
||||
| `--update` | Channel, frequency, pending updates |
|
||||
| `--all` | All of the above |
|
||||
|
||||
## Safety rules (MANDATORY)
|
||||
|
||||
- **NEVER hard-stop during first boot** -- corrupts TPM permanently,
|
||||
requires full reinstallation. Hard-stop is safe ONLY for already-
|
||||
provisioned nodes (TPM state already sealed).
|
||||
- **One node at a time** for break-fix exercises. 3-node cluster needs
|
||||
2/3 quorum. Taking down 2 nodes = cluster failure.
|
||||
- **Proxmox snapshots** before every destructive test (VMID range 900-939).
|
||||
- **Target node-03** (VMID 902) for destructive tests -- not the leader,
|
||||
fewest workloads.
|
||||
- **Verify health** with `incusos-health --all` before and after exercises.
|
||||
- **Monitor via Grafana** during all exercises.
|
||||
|
||||
## Update mechanics
|
||||
|
||||
- `os_version` = current running version.
|
||||
- `os_version_next` = version on standby partition (same as current if
|
||||
no update pending, different if update downloaded).
|
||||
- `needs_reboot` = true when update is downloaded and waiting for reboot.
|
||||
- `auto_reboot: false` in lab -- updates download but don't reboot.
|
||||
- Check frequency: 6 hours, stable channel.
|
||||
- Nodes update independently (rolling, not coordinated).
|
||||
|
||||
## Break-fix exercises (defined, not yet executed)
|
||||
|
||||
1. **Normal Update Observation** -- watch download, reboot, A/B switch
|
||||
2. **Simulated Failed Update** -- hard-stop node-03 mid-update, expect rollback
|
||||
3. **Network Isolation** -- disconnect node-03 NIC, observe OVN tunnel loss
|
||||
4. **Full Node Failure** -- hard-stop node-03, verify 2/3 quorum holds
|
||||
|
||||
See `notes/incusos-break-fix.md` for full exercise details, prerequisites,
|
||||
expected behavior, and recovery steps.
|
||||
|
|
@ -35,7 +35,8 @@ incus-contrib/
|
|||
│ │ ├── proxmox-api # Authenticated API calls (handles ! in token)
|
||||
│ │ ├── aether-browser # Playwright browser automation for Aether web UI
|
||||
│ │ ├── ovn-inspect # OVN/OVS topology inspection (--nb/--sb/--ovs/--trace)
|
||||
│ │ └── incus-mitm # API traffic capture and analysis (--capture/--live/--analyze)
|
||||
│ │ ├── incus-mitm # API traffic capture and analysis (--capture/--live/--analyze)
|
||||
│ │ └── incusos-health # IncusOS health inspection (--status/--tpm/--partitions/--all)
|
||||
│ ├── lab-test # Guided lab validation (12 test phases)
|
||||
│ ├── observe-deploy # Single-VM deploy with console screenshots
|
||||
│ ├── proxmox.yaml # Proxmox connection (gitignored)
|
||||
|
|
@ -192,6 +193,7 @@ which files are being edited:
|
|||
| `observability.md` | deploy-observability, observability dashboards/guide |
|
||||
| `ovn-internals.md` | ovn-inspect, ovn-deep-dive |
|
||||
| `aether-api-payloads.md` | incus-mitm, api-interception |
|
||||
| `incusos-immutability.md` | incusos-health, incusos-break-fix |
|
||||
| `proxmox-ssh-rules.md` | **Always loaded** (no paths filter) |
|
||||
| `lab-infrastructure.md` | incusos-proxmox, lab-test, examples |
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,758 @@
|
|||
#!/usr/bin/env bash
|
||||
# incusos-health -- Inspect IncusOS node health via the /os/1.0 REST API
|
||||
#
|
||||
# Queries one or more IncusOS cluster nodes for status, storage, security,
|
||||
# services, network, and update information.
|
||||
#
|
||||
# Usage: incusos-health [OPTIONS] [ACTION]
|
||||
# Run 'incusos-health --help' for full usage information.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
readonly VERSION="0.1.0"
|
||||
readonly SCRIPT_NAME="incusos-health"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Defaults
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
REMOTE="oc-node-01"
|
||||
NODE=""
|
||||
JSON_OUTPUT=false
|
||||
DRY_RUN=false
|
||||
VERBOSE=false
|
||||
QUIET=false
|
||||
ACTION=""
|
||||
|
||||
# Services to check (IncusOS built-in services)
|
||||
readonly SERVICES=(iscsi lvm multipath nvme ovn tailscale usbip)
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Color and formatting
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
setup_colors() {
|
||||
if [[ -t 1 ]] && [[ "${NO_COLOR:-}" != "1" ]] && [[ "${TERM:-}" != "dumb" ]]; then
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[0;33m'
|
||||
BLUE='\033[0;34m'
|
||||
CYAN='\033[0;36m'
|
||||
BOLD='\033[1m'
|
||||
DIM='\033[2m'
|
||||
RESET='\033[0m'
|
||||
else
|
||||
RED='' GREEN='' YELLOW='' BLUE='' CYAN='' BOLD='' DIM='' RESET=''
|
||||
fi
|
||||
}
|
||||
|
||||
info() { [[ "$QUIET" == true ]] && return; echo -e "${BLUE}[info]${RESET} $*"; }
|
||||
success() { [[ "$QUIET" == true ]] && return; echo -e "${GREEN}[ok]${RESET} $*"; }
|
||||
warn() { echo -e "${YELLOW}[warn]${RESET} $*" >&2; }
|
||||
error() { echo -e "${RED}[error]${RESET} $*" >&2; }
|
||||
step() { [[ "$QUIET" == true ]] && return; echo -e "${CYAN}[step]${RESET} ${BOLD}$*${RESET}"; }
|
||||
detail() { [[ "$VERBOSE" != true ]] && return; echo -e "${DIM} $*${RESET}"; }
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Help
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
usage() {
|
||||
cat <<EOF
|
||||
${BOLD}${SCRIPT_NAME}${RESET} v${VERSION} - Inspect IncusOS node health
|
||||
|
||||
${BOLD}USAGE${RESET}
|
||||
${SCRIPT_NAME} [OPTIONS] ACTION
|
||||
|
||||
${BOLD}ACTIONS${RESET}
|
||||
-s, --status Quick status overview (hostname, version, uptime,
|
||||
memory, TPM, Secure Boot, trusted state, update)
|
||||
-p, --partitions Storage and partition info (drives, ZFS pools,
|
||||
volumes, encryption status)
|
||||
-t, --tpm TPM and security details (TPM status, Secure Boot,
|
||||
certificates, recovery keys, trusted state)
|
||||
-S, --services Service configuration (iscsi, lvm, multipath, nvme,
|
||||
ovn, tailscale, usbip; OVN details)
|
||||
-N, --network Network configuration (interfaces, addresses, routes,
|
||||
state, DNS)
|
||||
-u, --update Update configuration and state (channel, frequency,
|
||||
auto reboot, versions, last check)
|
||||
-a, --all Run all of the above
|
||||
|
||||
${BOLD}OPTIONS${RESET}
|
||||
-r, --remote NAME Incus remote (default: ${REMOTE})
|
||||
--node NAME Specific node (default: all nodes in cluster)
|
||||
-j, --json Output raw JSON instead of formatted text
|
||||
-n, --dry-run Show what would be queried without executing
|
||||
-v, --verbose Show detailed output
|
||||
-q, --quiet Suppress informational output
|
||||
-h, --help Show this help message
|
||||
|
||||
${BOLD}API ENDPOINTS${RESET}
|
||||
/os/1.0 Environment (hostname, version, uptime)
|
||||
/os/1.0/system/security TPM, Secure Boot, encryption
|
||||
/os/1.0/system/storage Drives, ZFS pools, volumes
|
||||
/os/1.0/system/resources CPU, memory
|
||||
/os/1.0/system/network Interfaces, DNS
|
||||
/os/1.0/system/update Update config and state
|
||||
/os/1.0/services/SERVICE Per-service configuration
|
||||
|
||||
${BOLD}EXAMPLES${RESET}
|
||||
# Quick status of all cluster nodes
|
||||
${SCRIPT_NAME} --status
|
||||
|
||||
# Storage info for a specific node
|
||||
${SCRIPT_NAME} --partitions --node oc-node-02
|
||||
|
||||
# All info as JSON
|
||||
${SCRIPT_NAME} --all --json
|
||||
|
||||
# Different remote
|
||||
${SCRIPT_NAME} --status --remote lab-node-01
|
||||
|
||||
# Preview queries without running
|
||||
${SCRIPT_NAME} --all --dry-run
|
||||
EOF
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Argument parsing
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
parse_args() {
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
-s|--status) ACTION="status";;
|
||||
-p|--partitions) ACTION="partitions";;
|
||||
-t|--tpm) ACTION="tpm";;
|
||||
-S|--services) ACTION="services";;
|
||||
-N|--network) ACTION="network";;
|
||||
-u|--update) ACTION="update";;
|
||||
-a|--all) ACTION="all";;
|
||||
-r|--remote) REMOTE="$2"; shift;;
|
||||
--node) NODE="$2"; shift;;
|
||||
-j|--json) JSON_OUTPUT=true;;
|
||||
-n|--dry-run) DRY_RUN=true;;
|
||||
-v|--verbose) VERBOSE=true;;
|
||||
-q|--quiet) QUIET=true;;
|
||||
-h|--help) usage; exit 0;;
|
||||
*)
|
||||
error "Unknown option: $1"
|
||||
echo "Run '${SCRIPT_NAME} --help' for usage." >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
shift
|
||||
done
|
||||
|
||||
if [[ -z "$ACTION" ]]; then
|
||||
error "No action specified"
|
||||
echo "Run '${SCRIPT_NAME} --help' for usage." >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Node discovery
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
discover_nodes() {
|
||||
if [[ -n "$NODE" ]]; then
|
||||
echo "$NODE"
|
||||
return
|
||||
fi
|
||||
|
||||
local members
|
||||
members=$(incus cluster list "${REMOTE}:" -f csv 2>/dev/null \
|
||||
| cut -d',' -f1 \
|
||||
| sed 's/^ *//;s/ *$//') || true
|
||||
|
||||
if [[ -z "$members" ]]; then
|
||||
# Not a cluster or cannot list -- fall back to the remote itself
|
||||
echo "$REMOTE"
|
||||
return
|
||||
fi
|
||||
|
||||
echo "$members"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Query helper
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
query_node() {
|
||||
local node="$1" endpoint="$2"
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
info "(dry-run) incus query ${node}:${endpoint}"
|
||||
return 0
|
||||
fi
|
||||
|
||||
detail "incus query ${node}:${endpoint}"
|
||||
incus query "${node}:${endpoint}" 2>/dev/null || echo "{}"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Section: status
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
show_status() {
|
||||
local node="$1"
|
||||
|
||||
local env_json resources_json security_json update_json
|
||||
env_json=$(query_node "$node" "/os/1.0")
|
||||
resources_json=$(query_node "$node" "/os/1.0/system/resources")
|
||||
security_json=$(query_node "$node" "/os/1.0/system/security")
|
||||
update_json=$(query_node "$node" "/os/1.0/system/update")
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then return; fi
|
||||
|
||||
if [[ "$JSON_OUTPUT" == true ]]; then
|
||||
python3 -c "
|
||||
import json, sys
|
||||
print(json.dumps({
|
||||
'environment': json.loads(sys.argv[1]),
|
||||
'resources': json.loads(sys.argv[2]),
|
||||
'security': json.loads(sys.argv[3]),
|
||||
'update': json.loads(sys.argv[4]),
|
||||
}, indent=2))
|
||||
" "$env_json" "$resources_json" "$security_json" "$update_json"
|
||||
return
|
||||
fi
|
||||
|
||||
python3 -c "
|
||||
import json, sys
|
||||
|
||||
env = json.loads(sys.argv[1]).get('environment', json.loads(sys.argv[1]))
|
||||
res = json.loads(sys.argv[2])
|
||||
sec_raw = json.loads(sys.argv[3])
|
||||
sec = sec_raw.get('state', sec_raw)
|
||||
upd = json.loads(sys.argv[4])
|
||||
|
||||
hostname = env.get('hostname', 'unknown')
|
||||
version = env.get('os_version', env.get('version', 'unknown'))
|
||||
uptime_s = env.get('uptime', 0)
|
||||
|
||||
# Format uptime
|
||||
days = uptime_s // 86400
|
||||
hours = (uptime_s % 86400) // 3600
|
||||
mins = (uptime_s % 3600) // 60
|
||||
parts = []
|
||||
if days: parts.append(f'{days}d')
|
||||
if hours: parts.append(f'{hours}h')
|
||||
parts.append(f'{mins}m')
|
||||
uptime_str = ' '.join(parts)
|
||||
|
||||
# Memory
|
||||
mem = res.get('memory', {})
|
||||
mem_used = mem.get('used', 0)
|
||||
mem_total = mem.get('total', 0)
|
||||
|
||||
def fmt_bytes(b):
|
||||
for unit in ['B','KiB','MiB','GiB','TiB']:
|
||||
if b < 1024:
|
||||
return f'{b:.1f} {unit}'
|
||||
b /= 1024
|
||||
return f'{b:.1f} PiB'
|
||||
|
||||
mem_pct = (mem_used / mem_total * 100) if mem_total else 0
|
||||
|
||||
# Security (state level)
|
||||
tpm_status = sec.get('tpm_status', 'unknown')
|
||||
secboot_status = 'enabled' if sec.get('secure_boot_enabled', False) else 'disabled'
|
||||
trusted = sec.get('system_state_is_trusted', 'unknown')
|
||||
|
||||
# Update
|
||||
needs_reboot = upd.get('state', {}).get('needs_reboot', False)
|
||||
last_check = upd.get('state', {}).get('last_check', 'never')
|
||||
|
||||
print(f' Hostname: {hostname}')
|
||||
print(f' Version: {version}')
|
||||
print(f' Uptime: {uptime_str}')
|
||||
print(f' Memory: {fmt_bytes(mem_used)} / {fmt_bytes(mem_total)} ({mem_pct:.0f}%)')
|
||||
print(f' TPM: {tpm_status}')
|
||||
print(f' Secure Boot: {secboot_status}')
|
||||
print(f' System trusted: {trusted}')
|
||||
print(f' Needs reboot: {\"yes\" if needs_reboot else \"no\"}')
|
||||
print(f' Last update: {last_check}')
|
||||
" "$env_json" "$resources_json" "$security_json" "$update_json"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Section: partitions
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
show_partitions() {
|
||||
local node="$1"
|
||||
|
||||
local storage_json security_json
|
||||
storage_json=$(query_node "$node" "/os/1.0/system/storage")
|
||||
security_json=$(query_node "$node" "/os/1.0/system/security")
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then return; fi
|
||||
|
||||
if [[ "$JSON_OUTPUT" == true ]]; then
|
||||
python3 -c "
|
||||
import json, sys
|
||||
print(json.dumps({
|
||||
'storage': json.loads(sys.argv[1]),
|
||||
'security': json.loads(sys.argv[2]),
|
||||
}, indent=2))
|
||||
" "$storage_json" "$security_json"
|
||||
return
|
||||
fi
|
||||
|
||||
python3 -c "
|
||||
import json, sys
|
||||
|
||||
stor_raw = json.loads(sys.argv[1])
|
||||
sec_raw = json.loads(sys.argv[2])
|
||||
stor = stor_raw.get('state', stor_raw)
|
||||
sec = sec_raw.get('state', sec_raw)
|
||||
|
||||
def fmt_bytes(b):
|
||||
for unit in ['B','KiB','MiB','GiB','TiB']:
|
||||
if b < 1024:
|
||||
return f'{b:.1f} {unit}'
|
||||
b /= 1024
|
||||
return f'{b:.1f} PiB'
|
||||
|
||||
# Drives
|
||||
disks = stor.get('drives', stor.get('disks', []))
|
||||
if disks:
|
||||
print(' Drives:')
|
||||
for d in disks:
|
||||
model = d.get('model_name', d.get('model', 'unknown'))
|
||||
bus = d.get('bus', 'unknown')
|
||||
size = d.get('capacity_in_bytes', d.get('size', 0))
|
||||
boot = ' [boot]' if d.get('boot') else ''
|
||||
did = d.get('id', '?').split('/')[-1]
|
||||
pool = d.get('member_pool', '')
|
||||
pool_str = f' pool={pool}' if pool else ''
|
||||
print(f' {did}: {model} ({bus}, {fmt_bytes(size)}){boot}{pool_str}')
|
||||
else:
|
||||
print(' Drives: none found')
|
||||
|
||||
# ZFS pools
|
||||
pools = stor.get('pools', [])
|
||||
if pools:
|
||||
print(' ZFS pools:')
|
||||
for p in pools:
|
||||
name = p.get('name', 'unknown')
|
||||
state = p.get('state', 'unknown')
|
||||
ptype = p.get('type', 'unknown')
|
||||
alloc = p.get('pool_allocated_space_in_bytes', 0)
|
||||
total = p.get('raw_pool_size_in_bytes', p.get('usable_pool_size_in_bytes', 0))
|
||||
pct = (alloc / total * 100) if total else 0
|
||||
enc_key = p.get('encryption_key_status', 'N/A')
|
||||
print(f' {name}: {state} ({ptype}, {fmt_bytes(alloc)} / {fmt_bytes(total)}, {pct:.0f}%) enc_key={enc_key}')
|
||||
for v in p.get('volumes', []):
|
||||
vname = v.get('name', '?')
|
||||
vuse = v.get('use', '?')
|
||||
vusage = v.get('usage_in_bytes', 0)
|
||||
vquota = v.get('quota_in_bytes', 0)
|
||||
quota_str = fmt_bytes(vquota) if vquota else 'no quota'
|
||||
print(f' vol {vname} ({vuse}): {fmt_bytes(vusage)} / {quota_str}')
|
||||
else:
|
||||
print(' ZFS pools: none found')
|
||||
|
||||
# Encryption
|
||||
encrypted_vols = sec.get('encrypted_volumes', [])
|
||||
if encrypted_vols:
|
||||
print(' Encryption:')
|
||||
for v in encrypted_vols:
|
||||
vol = v.get('volume', '?')
|
||||
state = v.get('state', '?')
|
||||
print(f' {vol}: {state}')
|
||||
else:
|
||||
print(' Encryption: not configured')
|
||||
" "$storage_json" "$security_json"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Section: tpm
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
show_tpm() {
|
||||
local node="$1"
|
||||
|
||||
local security_json
|
||||
security_json=$(query_node "$node" "/os/1.0/system/security")
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then return; fi
|
||||
|
||||
if [[ "$JSON_OUTPUT" == true ]]; then
|
||||
python3 -c "
|
||||
import json, sys
|
||||
print(json.dumps(json.loads(sys.argv[1]), indent=2))
|
||||
" "$security_json"
|
||||
return
|
||||
fi
|
||||
|
||||
python3 -c "
|
||||
import json, sys
|
||||
|
||||
sec_raw = json.loads(sys.argv[1])
|
||||
sec = sec_raw.get('state', sec_raw)
|
||||
sec_cfg = sec_raw.get('config', {})
|
||||
|
||||
# TPM
|
||||
tpm_status = sec.get('tpm_status', 'unknown')
|
||||
print(f' TPM status: {tpm_status}')
|
||||
|
||||
# Secure Boot
|
||||
sb_enabled = sec.get('secure_boot_enabled', False)
|
||||
print(f' Secure Boot: {\"enabled\" if sb_enabled else \"disabled\"}')
|
||||
|
||||
# Certificates
|
||||
certs = sec.get('secure_boot_certificates', [])
|
||||
if certs:
|
||||
print(' Certificates:')
|
||||
for c in certs:
|
||||
ctype = c.get('type', '?')
|
||||
subject = c.get('subject', 'unknown')
|
||||
fp = c.get('fingerprint', '')
|
||||
fp_short = fp[:16] + '...' if len(fp) > 16 else fp
|
||||
print(f' {ctype:4s} {subject}')
|
||||
if fp_short:
|
||||
print(f' fingerprint: {fp_short}')
|
||||
|
||||
# Encrypted volumes
|
||||
enc_vols = sec.get('encrypted_volumes', [])
|
||||
if enc_vols:
|
||||
print(' Encrypted volumes:')
|
||||
for v in enc_vols:
|
||||
print(f' {v.get(\"volume\", \"?\")}: {v.get(\"state\", \"?\")}')
|
||||
|
||||
# Recovery keys
|
||||
recovery_keys = sec_cfg.get('encryption_recovery_keys', [])
|
||||
retrieved = sec.get('encryption_recovery_keys_retrieved', False)
|
||||
print(f' Recovery keys: {len(recovery_keys)} key(s), retrieved={retrieved}')
|
||||
|
||||
# Pool recovery keys
|
||||
pool_keys = sec.get('pool_recovery_keys', {})
|
||||
if pool_keys:
|
||||
print(f' Pool keys: {list(pool_keys.keys())}')
|
||||
|
||||
# System trusted
|
||||
trusted = sec.get('system_state_is_trusted', 'unknown')
|
||||
print(f' System trusted: {trusted}')
|
||||
" "$security_json"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Section: services
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
show_services() {
|
||||
local node="$1"
|
||||
|
||||
if [[ "$JSON_OUTPUT" == true ]]; then
|
||||
local all_json="{"
|
||||
local first=true
|
||||
for svc in "${SERVICES[@]}"; do
|
||||
local svc_json
|
||||
svc_json=$(query_node "$node" "/os/1.0/services/${svc}")
|
||||
if [[ "$DRY_RUN" == true ]]; then continue; fi
|
||||
if [[ "$first" == true ]]; then
|
||||
first=false
|
||||
else
|
||||
all_json+=","
|
||||
fi
|
||||
all_json+="\"${svc}\":${svc_json}"
|
||||
done
|
||||
all_json+="}"
|
||||
if [[ "$DRY_RUN" != true ]]; then
|
||||
echo "$all_json" | python3 -m json.tool
|
||||
fi
|
||||
return
|
||||
fi
|
||||
|
||||
for svc in "${SERVICES[@]}"; do
|
||||
local svc_json
|
||||
svc_json=$(query_node "$node" "/os/1.0/services/${svc}")
|
||||
if [[ "$DRY_RUN" == true ]]; then continue; fi
|
||||
|
||||
python3 -c "
|
||||
import json, sys
|
||||
|
||||
svc_name = sys.argv[1]
|
||||
data = json.loads(sys.argv[2])
|
||||
|
||||
config = data.get('config', data)
|
||||
enabled = config.get('enabled', False)
|
||||
status = 'enabled' if enabled else 'disabled'
|
||||
|
||||
line = f' {svc_name:12s} {status}'
|
||||
print(line)
|
||||
|
||||
# Extra details for OVN
|
||||
if svc_name == 'ovn' and enabled:
|
||||
db = config.get('database', '')
|
||||
tunnel_addr = config.get('tunnel_address', '')
|
||||
tunnel_proto = config.get('tunnel_protocol', '')
|
||||
if db:
|
||||
print(f' database: {db}')
|
||||
if tunnel_addr:
|
||||
print(f' tunnel address: {tunnel_addr}')
|
||||
if tunnel_proto:
|
||||
print(f' tunnel protocol: {tunnel_proto}')
|
||||
" "$svc" "$svc_json"
|
||||
done
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Section: network
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
show_network() {
|
||||
local node="$1"
|
||||
|
||||
local network_json
|
||||
network_json=$(query_node "$node" "/os/1.0/system/network")
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then return; fi
|
||||
|
||||
if [[ "$JSON_OUTPUT" == true ]]; then
|
||||
echo "$network_json" | python3 -m json.tool
|
||||
return
|
||||
fi
|
||||
|
||||
python3 -c "
|
||||
import json, sys
|
||||
|
||||
net = json.loads(sys.argv[1])
|
||||
|
||||
def fmt_bytes(b):
|
||||
for unit in ['B','KiB','MiB','GiB','TiB']:
|
||||
if b < 1024: return f'{b:.1f} {unit}'
|
||||
b /= 1024
|
||||
return f'{b:.1f} PiB'
|
||||
|
||||
# Interfaces
|
||||
ifaces = net.get('interfaces', net.get('config', {}).get('interfaces', []))
|
||||
if not isinstance(ifaces, list): ifaces = []
|
||||
print(' Interfaces:')
|
||||
for iface in ifaces:
|
||||
name = iface.get('name', 'unknown')
|
||||
hwaddr = iface.get('hwaddr', iface.get('mac', ''))
|
||||
roles = iface.get('roles', [])
|
||||
roles_str = ', '.join(roles) if isinstance(roles, list) and roles else ''
|
||||
addrs = iface.get('addresses', [])
|
||||
addr_strs = [a if isinstance(a, str) else a.get('address', '?') for a in addrs]
|
||||
state = iface.get('state', {})
|
||||
|
||||
header = f' {name}'
|
||||
if hwaddr: header += f' ({hwaddr})'
|
||||
if roles_str: header += f' [{roles_str}]'
|
||||
print(header)
|
||||
|
||||
if addr_strs:
|
||||
print(f' addresses: {\", \".join(addr_strs)}')
|
||||
|
||||
link_state = state.get('state', state.get('link', ''))
|
||||
mtu = state.get('mtu', iface.get('mtu', ''))
|
||||
speed = state.get('speed', '')
|
||||
if link_state:
|
||||
parts = [f'state={link_state}']
|
||||
if mtu: parts.append(f'mtu={mtu}')
|
||||
if speed: parts.append(f'speed={speed}')
|
||||
print(f' link: {\", \".join(parts)}')
|
||||
|
||||
routes = iface.get('routes', [])
|
||||
if isinstance(routes, list) and routes:
|
||||
route_strs = []
|
||||
for r in routes:
|
||||
if isinstance(r, str): route_strs.append(r)
|
||||
elif isinstance(r, dict):
|
||||
s = r.get('destination', r.get('to', '?'))
|
||||
via = r.get('via', r.get('gateway', ''))
|
||||
if via: s += f' via {via}'
|
||||
route_strs.append(s)
|
||||
if route_strs:
|
||||
print(f' routes: {route_strs[0]}')
|
||||
for rs in route_strs[1:]:
|
||||
print(f' {rs}')
|
||||
|
||||
stats = state.get('stats', iface.get('stats', {}))
|
||||
if isinstance(stats, dict) and stats:
|
||||
rx = stats.get('rx_bytes', stats.get('rx', 0))
|
||||
tx = stats.get('tx_bytes', stats.get('tx', 0))
|
||||
if rx or tx:
|
||||
print(f' stats: rx={fmt_bytes(rx)}, tx={fmt_bytes(tx)}')
|
||||
|
||||
# DNS
|
||||
dns = net.get('dns', net.get('config', {}).get('dns', {}))
|
||||
if isinstance(dns, dict) and dns:
|
||||
print(' DNS:')
|
||||
hostname = dns.get('hostname', dns.get('search', ''))
|
||||
nameservers = dns.get('nameservers', dns.get('servers', []))
|
||||
if hostname: print(f' hostname: {hostname}')
|
||||
if isinstance(nameservers, list) and nameservers:
|
||||
print(f' nameservers: {\", \".join(str(n) for n in nameservers)}')
|
||||
" "$network_json"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Section: update
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
show_update() {
|
||||
local node="$1"
|
||||
|
||||
local update_json env_json
|
||||
update_json=$(query_node "$node" "/os/1.0/system/update")
|
||||
env_json=$(query_node "$node" "/os/1.0")
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then return; fi
|
||||
|
||||
if [[ "$JSON_OUTPUT" == true ]]; then
|
||||
echo "$update_json" | python3 -m json.tool
|
||||
return
|
||||
fi
|
||||
|
||||
python3 -c "
|
||||
import json, sys
|
||||
|
||||
upd = json.loads(sys.argv[1])
|
||||
env = json.loads(sys.argv[2]).get('environment', json.loads(sys.argv[2]))
|
||||
config = upd.get('config', {})
|
||||
state = upd.get('state', {})
|
||||
|
||||
channel = config.get('channel', 'unknown')
|
||||
frequency = config.get('check_frequency', 'unknown')
|
||||
auto_reboot = config.get('auto_reboot', 'unknown')
|
||||
|
||||
current = env.get('os_version', 'unknown')
|
||||
next_ver = env.get('os_version_next', 'none')
|
||||
update_status = state.get('status', '')
|
||||
last_check = state.get('last_check', 'never')
|
||||
needs_reboot = state.get('needs_reboot', False)
|
||||
|
||||
print(f' Channel: {channel}')
|
||||
print(f' Check freq: {frequency}')
|
||||
print(f' Auto reboot: {auto_reboot}')
|
||||
print(f' Current: {current}')
|
||||
print(f' Next version: {next_ver}')
|
||||
print(f' Update status: {update_status}')
|
||||
print(f' Last check: {last_check}')
|
||||
print(f' Needs reboot: {\"yes\" if needs_reboot else \"no\"}')
|
||||
" "$update_json" "$env_json"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Run a section across all nodes
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
run_section() {
|
||||
local section_name="$1"
|
||||
local section_fn="$2"
|
||||
shift 2
|
||||
local nodes=("$@")
|
||||
|
||||
step "$section_name"
|
||||
|
||||
for node in "${nodes[@]}"; do
|
||||
echo -e " ${BOLD}--- ${node} ---${RESET}"
|
||||
"$section_fn" "$node"
|
||||
echo
|
||||
done
|
||||
}
|
||||
|
||||
run_section_json() {
|
||||
local section_fn="$1"
|
||||
shift
|
||||
local nodes=("$@")
|
||||
|
||||
if [[ ${#nodes[@]} -eq 1 ]]; then
|
||||
"$section_fn" "${nodes[0]}"
|
||||
else
|
||||
echo "{"
|
||||
local first=true
|
||||
for node in "${nodes[@]}"; do
|
||||
if [[ "$first" == true ]]; then
|
||||
first=false
|
||||
else
|
||||
echo ","
|
||||
fi
|
||||
echo " \"${node}\":"
|
||||
"$section_fn" "$node" | sed 's/^/ /'
|
||||
done
|
||||
echo "}"
|
||||
fi
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Main
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
main() {
|
||||
setup_colors
|
||||
parse_args "$@"
|
||||
|
||||
# Discover cluster nodes
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
if [[ -n "$NODE" ]]; then
|
||||
info "(dry-run) Would query node: ${NODE}"
|
||||
else
|
||||
info "(dry-run) Would discover nodes via: incus cluster list ${REMOTE}: -f csv"
|
||||
fi
|
||||
fi
|
||||
|
||||
local nodes_raw
|
||||
nodes_raw=$(discover_nodes)
|
||||
|
||||
local nodes=()
|
||||
while IFS= read -r line; do
|
||||
[[ -n "$line" ]] && nodes+=("$line")
|
||||
done <<< "$nodes_raw"
|
||||
|
||||
if [[ ${#nodes[@]} -eq 0 ]]; then
|
||||
error "No nodes found. Check that remote '${REMOTE}' is accessible."
|
||||
error " incus remote list"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ "$DRY_RUN" != true ]]; then
|
||||
info "Nodes: ${nodes[*]}"
|
||||
fi
|
||||
echo
|
||||
|
||||
# Section lookup: action -> (title, function)
|
||||
local -A SECTION_TITLE=(
|
||||
[status]="Status" [partitions]="Storage & Partitions"
|
||||
[tpm]="TPM & Security" [services]="Services"
|
||||
[network]="Network" [update]="Update"
|
||||
)
|
||||
|
||||
# Build section list
|
||||
local sections=()
|
||||
if [[ "$ACTION" == "all" ]]; then
|
||||
sections=(status partitions tpm services network update)
|
||||
else
|
||||
sections=("$ACTION")
|
||||
fi
|
||||
|
||||
# Dispatch
|
||||
if [[ "$JSON_OUTPUT" == true && ${#sections[@]} -gt 1 ]]; then
|
||||
echo "{"
|
||||
local i=0
|
||||
for section in "${sections[@]}"; do
|
||||
echo " \"${section}\":"
|
||||
run_section_json "show_${section}" "${nodes[@]}" | sed 's/^/ /'
|
||||
i=$((i + 1))
|
||||
[[ $i -lt ${#sections[@]} ]] && echo ","
|
||||
done
|
||||
echo "}"
|
||||
else
|
||||
for section in "${sections[@]}"; do
|
||||
if [[ "$JSON_OUTPUT" == true ]]; then
|
||||
run_section_json "show_${section}" "${nodes[@]}"
|
||||
else
|
||||
run_section "${SECTION_TITLE[$section]}" "show_${section}" "${nodes[@]}"
|
||||
fi
|
||||
done
|
||||
fi
|
||||
}
|
||||
|
||||
main "$@"
|
||||
|
|
@ -0,0 +1,421 @@
|
|||
# IncusOS Break-Fix Lab -- Immutability Exploration & Resilience Testing
|
||||
|
||||
IncusOS is an immutable, purpose-built operating system for running Incus
|
||||
clusters. This guide documents what we discovered about its internals through
|
||||
the `/os/1.0` API, and defines a catalog of break-fix exercises for testing
|
||||
cluster resilience in a safe lab environment.
|
||||
|
||||
All observations come from a 3-node Proxmox-hosted cluster (VMID 900-902)
|
||||
running IncusOS with virtual TPM, Secure Boot, and OVN networking.
|
||||
|
||||
## IncusOS Architecture
|
||||
|
||||
IncusOS is an immutable OS designed for a single purpose: running Incus.
|
||||
Key architectural properties discovered via the `/os/1.0` API:
|
||||
|
||||
- **Immutable root filesystem** with A/B partition scheme. The running system
|
||||
boots from one partition; updates install to the other. Rollback is automatic
|
||||
if the new partition fails validation.
|
||||
- **TPM-based full disk encryption** -- root and swap volumes are encrypted
|
||||
and unlocked automatically via TPM measured boot. No passphrase required
|
||||
during normal operation.
|
||||
- **Secure Boot enforced** with 4 certificates in the UEFI firmware:
|
||||
- PK (Platform Key) -- root of trust
|
||||
- KEK (Key Exchange Key) -- authorizes db updates
|
||||
- db (2025 signing cert) -- validates current signed binaries
|
||||
- db (2026 signing cert) -- validates next-year signed binaries
|
||||
- **Version format**: `YYYYMMDDHHMI` (build timestamp, not semver).
|
||||
Example: `202602240349` = 2026-02-24 at 03:49 UTC.
|
||||
- **ZFS storage**: pool "local" on a dedicated partition (raid0 on single
|
||||
disk, ~30 GiB usable), encrypted with its own pool recovery key.
|
||||
- **Update system**: stable channel, 6-hour check frequency,
|
||||
`auto_reboot: false` (updates download but do not reboot automatically).
|
||||
|
||||
### Lab cluster state
|
||||
|
||||
| Node | VMID | IP | IncusOS Version |
|
||||
|------|------|----|-----------------|
|
||||
| node-01 | 900 | 192.168.102.140 | 202602240349 |
|
||||
| node-02 | 901 | 192.168.102.141 | 202602230420 |
|
||||
| node-03 | 902 | 192.168.102.142 | 202602230420 |
|
||||
|
||||
node-01 received a newer build than nodes 02/03, proving that nodes update
|
||||
independently (rolling updates, not cluster-wide atomic upgrades).
|
||||
|
||||
## Partition & Disk Layout
|
||||
|
||||
Each node has a single 64 GiB SCSI disk (QEMU HARDDISK) with this layout:
|
||||
|
||||
```
|
||||
+------------------------------------------------------+
|
||||
| SCSI Disk (64 GiB, QEMU HARDDISK) |
|
||||
|------------------------------------------------------|
|
||||
| Partition 1 EFI System Partition |
|
||||
| Partition A Root filesystem (active or standby) |
|
||||
| Partition B Root filesystem (standby or active) |
|
||||
| Partition X Swap (encrypted, TPM-unlocked) |
|
||||
| Partition 11 ZFS data pool ("local") |
|
||||
+------------------------------------------------------+
|
||||
```
|
||||
|
||||
Key observations from the storage API:
|
||||
|
||||
- **Partition 11** is the ZFS data partition, hosting pool "local".
|
||||
- **Pool "local"**: raid0, single vdev, ~30 GiB total, contains volume
|
||||
"incus" (~10 GiB used for Incus database, images, instances).
|
||||
- **Root + swap**: both encrypted, both unlocked by TPM at boot.
|
||||
No manual key entry needed unless TPM state is corrupted.
|
||||
|
||||
## Security Chain
|
||||
|
||||
The boot security chain, as validated through the API:
|
||||
|
||||
```
|
||||
UEFI firmware
|
||||
--> Secure Boot certificate validation (PK -> KEK -> db certs)
|
||||
--> Signed kernel + initrd loaded
|
||||
--> TPM measured boot (PCR measurements recorded)
|
||||
--> TPM validates measurements match expected policy
|
||||
--> Root partition decrypted and mounted
|
||||
--> Swap partition decrypted and activated
|
||||
--> System boots into trusted state
|
||||
```
|
||||
|
||||
API-reported security state:
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| TPM status | ok |
|
||||
| Secure Boot | enabled, enforced |
|
||||
| System state | trusted |
|
||||
| Encryption recovery keys | 1 key (retrievable via API) |
|
||||
| Pool recovery keys | 1 key for pool "local" |
|
||||
|
||||
**Recovery keys** are a safety net. If TPM state is corrupted (e.g., by a
|
||||
hard-stop during first boot), the encryption recovery key allows manual
|
||||
unlock. The pool recovery key allows ZFS pool import on a different system.
|
||||
|
||||
## Services Configuration
|
||||
|
||||
Discovered via `/os/1.0/services/*`:
|
||||
|
||||
| Service | Status | Notes |
|
||||
|---------|--------|-------|
|
||||
| OVN | enabled | Connects to cluster DB at `tcp:192.168.102.142:6642`, Geneve tunnels |
|
||||
| iSCSI | disabled | |
|
||||
| LVM | disabled | |
|
||||
| Multipath | disabled | |
|
||||
| NVMe | disabled | |
|
||||
| Tailscale | disabled | |
|
||||
| USB/IP | disabled | |
|
||||
|
||||
OVN is the only enabled service beyond Incus itself, providing the overlay
|
||||
network for cross-node container connectivity and network policies.
|
||||
|
||||
## Network Configuration
|
||||
|
||||
Each node has a single management interface:
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| Interface name | mgmt |
|
||||
| Role | management + cluster traffic |
|
||||
| Addressing | Static |
|
||||
| Subnet | /22 (192.168.100.0/22) |
|
||||
| DNS | 192.168.100.1 |
|
||||
| Gateway | 192.168.100.1 |
|
||||
|
||||
Node IP assignments:
|
||||
- node-01: `192.168.102.140/22`
|
||||
- node-02: `192.168.102.141/22`
|
||||
- node-03: `192.168.102.142/22`
|
||||
|
||||
The management interface carries both management traffic (API, cluster
|
||||
heartbeats) and OVN Geneve tunnel traffic. In production, these should
|
||||
be separated.
|
||||
|
||||
## Update Mechanics (Observed)
|
||||
|
||||
The IncusOS update system uses A/B partitions for safe, rollback-capable
|
||||
updates. Observations from querying the update API:
|
||||
|
||||
1. **No pending update**: `os_version` and `os_version_next` are identical.
|
||||
This means no update has been downloaded or is waiting for reboot.
|
||||
|
||||
2. **Pending update**: `os_version_next` would differ from `os_version`,
|
||||
indicating a new build has been downloaded to the standby partition.
|
||||
|
||||
3. **Auto-reboot disabled**: `auto_reboot: false` means updates download
|
||||
to the standby partition but the node continues running the current
|
||||
version until explicitly rebooted (or until an admin triggers reboot
|
||||
via API / UI).
|
||||
|
||||
4. **No reboot needed**: `needs_reboot: false` confirms no downloaded
|
||||
update is waiting for a reboot to activate.
|
||||
|
||||
5. **Independent node updates**: node-01 is on `202602240349` while
|
||||
nodes 02/03 are on `202602230420`. This proves each node checks for
|
||||
and applies updates independently. There is no cluster-wide coordinated
|
||||
update mechanism at the OS level.
|
||||
|
||||
6. **Check frequency**: 6 hours. The stable channel is checked
|
||||
automatically on this interval.
|
||||
|
||||
### Update lifecycle (theoretical)
|
||||
|
||||
```
|
||||
Check timer fires (every 6h)
|
||||
--> Query stable channel for new version
|
||||
--> Download new rootfs to standby partition (A or B)
|
||||
--> os_version_next updated, needs_reboot = true
|
||||
--> (if auto_reboot) Reboot automatically
|
||||
--> (if !auto_reboot) Wait for manual reboot
|
||||
--> On reboot: boot from new partition
|
||||
--> TPM re-measures, validates new boot chain
|
||||
--> If valid: new partition becomes active
|
||||
--> If invalid: rollback to previous partition
|
||||
```
|
||||
|
||||
## Break-Fix Exercise Catalog
|
||||
|
||||
These exercises are designed to test IncusOS cluster resilience in the
|
||||
Proxmox lab. All exercises follow strict safety rules (see Safety Rules
|
||||
section below).
|
||||
|
||||
**Current status**: All exercises are defined but not yet executed.
|
||||
|
||||
---
|
||||
|
||||
### Exercise 1: Normal Update Observation
|
||||
|
||||
**Goal**: Observe the full update lifecycle -- download, reboot, A/B
|
||||
partition switch, version verification.
|
||||
|
||||
**Prerequisites**:
|
||||
- Proxmox snapshot of all cluster nodes (VMID 900-902)
|
||||
- Grafana monitoring active (observe cluster metrics during update)
|
||||
- Verify cluster health with `incusos-health --all`
|
||||
|
||||
**Steps**:
|
||||
1. Record current versions: `incusos-health --update`
|
||||
2. Trigger update via Operations Center UI, or wait for 6h check interval
|
||||
3. Monitor via Grafana for download activity and node state changes
|
||||
4. When `needs_reboot: true`, reboot one node at a time
|
||||
5. After reboot, verify new version: `incusos-health --status`
|
||||
6. Confirm A/B partition switch: `incusos-health --partitions`
|
||||
|
||||
**What to observe**:
|
||||
- Download phase duration
|
||||
- Reboot duration (typically 30-60s for IncusOS)
|
||||
- Cluster behavior while one node reboots (2/3 quorum maintained)
|
||||
- New version number in `os_version`
|
||||
- Previous version still available on standby partition
|
||||
|
||||
**Status**: Defined, not yet executed.
|
||||
|
||||
---
|
||||
|
||||
### Exercise 2: Simulated Failed Update (Hard-Stop Mid-Update)
|
||||
|
||||
**Goal**: Verify that IncusOS rolls back to the previous partition after
|
||||
a failed update (simulated by hard-stopping the VM during update).
|
||||
|
||||
**Prerequisites**:
|
||||
- Proxmox snapshot of node-03 (VMID 902) -- non-leader, fewest workloads
|
||||
- Verify node-03 has no critical workloads
|
||||
- Confirm 3/3 nodes healthy before starting
|
||||
|
||||
**Target node**: node-03 (VMID 902) ONLY. Never the cluster leader.
|
||||
|
||||
**Steps**:
|
||||
1. Take Proxmox snapshot of node-03: `incusos/helpers/proxmox-api POST /nodes/pve/qemu/902/snapshot -d '{"snapname":"pre-break-fix-2"}'`
|
||||
2. Trigger an update on node-03 (if one is available)
|
||||
3. During the update download or early reboot phase, hard-stop node-03
|
||||
via Proxmox API: `incusos/helpers/proxmox-api POST /nodes/pve/qemu/902/status/stop`
|
||||
4. Wait 10 seconds, then start node-03:
|
||||
`incusos/helpers/proxmox-api POST /nodes/pve/qemu/902/status/start`
|
||||
5. Monitor boot via console screenshots: `incusos/helpers/proxmox-screenshot 902`
|
||||
6. After boot, check version: `incusos-health --status` on node-03
|
||||
|
||||
**Expected behavior**:
|
||||
- Node boots from the previous (known-good) partition
|
||||
- Failed partition is marked as bad / not bootable
|
||||
- `os_version` returns to the pre-update version
|
||||
- Node rejoins the cluster automatically
|
||||
|
||||
**Safety**:
|
||||
- ONLY node-03 -- maintains 2/3 quorum (node-01 + node-02 continue)
|
||||
- Do NOT hard-stop during *first boot* (corrupts TPM permanently)
|
||||
- If recovery fails, restore from Proxmox snapshot
|
||||
|
||||
**Status**: Defined, not yet executed.
|
||||
|
||||
---
|
||||
|
||||
### Exercise 3: Network Isolation
|
||||
|
||||
**Goal**: Observe how the cluster handles a node losing network
|
||||
connectivity -- OVN tunnel loss, cluster membership changes, and
|
||||
automatic recovery on reconnection.
|
||||
|
||||
**Prerequisites**:
|
||||
- Evacuate all workloads from node-03 before disconnecting
|
||||
- Grafana monitoring active (watch OVN tunnel metrics, cluster events)
|
||||
- Verify cluster health: `incusos-health --all`
|
||||
|
||||
**Steps**:
|
||||
1. Evacuate workloads from node-03:
|
||||
`incus cluster evacuate oc-node-03 --target oc-node-01`
|
||||
2. Disconnect node-03 NIC via Proxmox API:
|
||||
`incusos/helpers/proxmox-api PUT /nodes/pve/qemu/902/config -d '{"net0":"virtio=...,link_down=1"}'`
|
||||
3. Observe in Grafana: OVN Geneve tunnel loss, cluster detecting missing node
|
||||
4. Wait 2-5 minutes for cluster to mark node-03 as offline
|
||||
5. Reconnect NIC: set `link_down=0` via Proxmox API
|
||||
6. Monitor recovery: node-03 should rejoin cluster, OVN tunnels re-establish
|
||||
7. Measure total recovery time from reconnection to healthy state
|
||||
|
||||
**Expected behavior**:
|
||||
- Cluster detects node-03 offline within heartbeat timeout
|
||||
- OVN tunnels from/to node-03 fail (Geneve encap packets lost)
|
||||
- Cluster continues operating with 2/3 quorum
|
||||
- On reconnection: node-03 rejoins, tunnels re-establish, workloads
|
||||
can be restored
|
||||
|
||||
**Recovery**:
|
||||
- If node-03 does not rejoin: check OVN service status, restart if needed
|
||||
- If cluster state is inconsistent: restore from Proxmox snapshot
|
||||
|
||||
**Status**: Defined, not yet executed.
|
||||
|
||||
---
|
||||
|
||||
### Exercise 4: Full Node Failure
|
||||
|
||||
**Goal**: Verify that the cluster survives a complete node loss and
|
||||
maintains operations with 2/3 quorum. Measure cluster rejoin time
|
||||
after node recovery.
|
||||
|
||||
**Prerequisites**:
|
||||
- Proxmox snapshot of node-03 (VMID 902)
|
||||
- Evacuate ALL workloads from node-03
|
||||
- Verify cluster health: `incusos-health --all`
|
||||
|
||||
**Steps**:
|
||||
1. Evacuate workloads: `incus cluster evacuate oc-node-03`
|
||||
2. Take Proxmox snapshot of node-03
|
||||
3. Hard-stop node-03 via Proxmox:
|
||||
`incusos/helpers/proxmox-api POST /nodes/pve/qemu/902/status/stop`
|
||||
4. Verify cluster continues operating:
|
||||
- `incus cluster list` should show 2 online, 1 offline
|
||||
- Existing workloads on node-01/02 remain accessible
|
||||
- OVN gateway should failover if node-03 was a gateway
|
||||
5. Wait 5 minutes, observing Grafana metrics
|
||||
6. Restart node-03:
|
||||
`incusos/helpers/proxmox-api POST /nodes/pve/qemu/902/status/start`
|
||||
7. Monitor boot: `incusos/helpers/proxmox-screenshot 902`
|
||||
8. Measure time from start to cluster rejoin (node shows online)
|
||||
9. Restore workloads: `incus cluster restore oc-node-03`
|
||||
|
||||
**Expected behavior**:
|
||||
- 2/3 quorum maintained -- all cluster operations continue
|
||||
- OVN gateway failover occurs if node-03 was elected gateway
|
||||
- After restart: node-03 boots, TPM unlocks, Incus starts, rejoins cluster
|
||||
- Typical rejoin time: 1-3 minutes after boot
|
||||
|
||||
**Safety**:
|
||||
- This is NOT a first-boot scenario -- hard-stop is safe for already-
|
||||
provisioned nodes (TPM state is already sealed)
|
||||
- One node at a time ONLY
|
||||
- If node-03 fails to rejoin after restart, restore snapshot
|
||||
|
||||
**Status**: Defined, not yet executed.
|
||||
|
||||
## Safety Rules
|
||||
|
||||
These rules are non-negotiable for all break-fix exercises:
|
||||
|
||||
1. **Never hard-stop during first boot.** First boot seals TPM measurements
|
||||
and writes encryption keys. Interrupting this corrupts TPM state
|
||||
permanently, requiring full reinstallation.
|
||||
|
||||
2. **One node at a time.** A 3-node cluster requires 2/3 quorum. Never
|
||||
take down more than one node simultaneously, or the cluster loses
|
||||
quorum and all operations halt.
|
||||
|
||||
3. **Proxmox snapshots before every destructive test.** Snapshot the
|
||||
target node's VM (VMID 900-939) before any exercise that involves
|
||||
stopping, disconnecting, or modifying the node.
|
||||
|
||||
4. **Verify cluster health before and after.** Run `incusos-health --all`
|
||||
(or equivalent API checks) before starting an exercise and after
|
||||
completing it. Do not proceed if the cluster is already degraded.
|
||||
|
||||
5. **Monitor via Grafana during all exercises.** Visual monitoring catches
|
||||
issues that API polling might miss (e.g., OVN tunnel flapping,
|
||||
storage I/O spikes).
|
||||
|
||||
6. **Target node-03 for destructive tests.** node-03 (VMID 902) is the
|
||||
preferred target because it is not the cluster leader and typically
|
||||
has the fewest workloads. Evacuate before testing.
|
||||
|
||||
7. **Keep recovery keys accessible.** The encryption recovery key and
|
||||
ZFS pool recovery key should be retrieved and stored securely before
|
||||
any exercise that might corrupt TPM state.
|
||||
|
||||
## Helper Script: incusos-health
|
||||
|
||||
The `incusos/helpers/incusos-health` script queries the IncusOS API on
|
||||
cluster nodes to report system state. It is the primary tool for verifying
|
||||
cluster health before and after break-fix exercises.
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
incusos/helpers/incusos-health [ACTION] [OPTIONS]
|
||||
```
|
||||
|
||||
### Actions
|
||||
|
||||
| Action | Description |
|
||||
|--------|-------------|
|
||||
| `--status` | Basic system info: version, hostname, TPM status, Secure Boot |
|
||||
| `--partitions` | Disk and partition layout, A/B partition state |
|
||||
| `--tpm` | TPM details, Secure Boot certificates, encryption keys |
|
||||
| `--services` | Enabled/disabled services (OVN, iSCSI, LVM, etc.) |
|
||||
| `--network` | Network interface configuration, DNS, gateway |
|
||||
| `--update` | Update channel, check frequency, pending update status |
|
||||
| `--all` | Run all of the above in sequence |
|
||||
|
||||
### Example output workflow
|
||||
|
||||
```bash
|
||||
# Before exercise: verify all nodes healthy
|
||||
incusos/helpers/incusos-health --all
|
||||
|
||||
# After exercise: verify recovery
|
||||
incusos/helpers/incusos-health --status # Quick version check
|
||||
incusos/helpers/incusos-health --all # Full health report
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
All IncusOS system information is available via the REST API on each node.
|
||||
|
||||
| Endpoint | Returns |
|
||||
|----------|---------|
|
||||
| `/os/1.0` | Version, hostname, basic system info |
|
||||
| `/os/1.0/system/security` | TPM status, Secure Boot, encryption keys |
|
||||
| `/os/1.0/system/storage` | Disks, partitions, ZFS pools |
|
||||
| `/os/1.0/system/resources` | CPU, memory, hardware info |
|
||||
| `/os/1.0/system/network` | Interfaces, DNS, routes |
|
||||
| `/os/1.0/system/update` | Update channel, version, pending updates |
|
||||
| `/os/1.0/services/ovn` | OVN configuration and status |
|
||||
| `/os/1.0/services/iscsi` | iSCSI configuration |
|
||||
| `/os/1.0/services/lvm` | LVM configuration |
|
||||
| `/os/1.0/services/multipath` | Multipath configuration |
|
||||
| `/os/1.0/services/nvme` | NVMe-oF configuration |
|
||||
| `/os/1.0/services/tailscale` | Tailscale VPN configuration |
|
||||
| `/os/1.0/services/usbip` | USB/IP configuration |
|
||||
|
||||
The API listens on the management interface, HTTPS, port 8443 (same as
|
||||
Incus). Authentication uses the Incus client certificate.
|
||||
Loading…
Reference in New Issue