Add IncusOS break-fix guide, incusos-health helper, immutability rules
Phase 4: Explore IncusOS immutability via the /os/1.0 API: - incusos-health script with --status/--partitions/--tpm/--services/--network/--update - Validated: TPM encryption (root+swap), Secure Boot with 4 certs, ZFS pool - Documented A/B partition scheme, update mechanics, security chain - Defined 4 break-fix exercises (update observation, failed update, network isolation, full node failure) with safety rules and pickup points Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
cc017b648d
commit
95316a5bee
|
|
@ -0,0 +1,88 @@
|
||||||
|
---
|
||||||
|
paths:
|
||||||
|
- "incusos/helpers/incusos-health"
|
||||||
|
- "notes/incusos-break-fix.md"
|
||||||
|
---
|
||||||
|
|
||||||
|
# IncusOS Immutability & Break-Fix Context
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
- IncusOS is an immutable OS with A/B partition scheme for safe updates.
|
||||||
|
- Version format: `YYYYMMDDHHMI` (build timestamp, e.g., `202602240349`).
|
||||||
|
- TPM-based full disk encryption: root + swap volumes unlocked via TPM
|
||||||
|
measured boot. Secure Boot enforced with PK, KEK, and 2x db certs.
|
||||||
|
- ZFS storage pool "local" on partition 11, contains volume "incus".
|
||||||
|
- Single SCSI disk per node (64 GiB in lab, QEMU HARDDISK).
|
||||||
|
|
||||||
|
## Lab cluster nodes
|
||||||
|
|
||||||
|
| Node | VMID | IP | Version |
|
||||||
|
|------|------|----|---------|
|
||||||
|
| node-01 | 900 | 192.168.102.140 | 202602240349 |
|
||||||
|
| node-02 | 901 | 192.168.102.141 | 202602230420 |
|
||||||
|
| node-03 | 902 | 192.168.102.142 | 202602230420 |
|
||||||
|
|
||||||
|
- Management + cluster traffic on interface "mgmt" (static IPs, /22 subnet).
|
||||||
|
- DNS + gateway: 192.168.100.1.
|
||||||
|
- OVN enabled, all other services disabled.
|
||||||
|
|
||||||
|
## IncusOS API endpoints
|
||||||
|
|
||||||
|
All on port 8443, authenticated with Incus client certificate:
|
||||||
|
|
||||||
|
- `/os/1.0` -- version, hostname, basic info
|
||||||
|
- `/os/1.0/system/security` -- TPM, Secure Boot, encryption/pool recovery keys
|
||||||
|
- `/os/1.0/system/storage` -- disks, partitions, ZFS pools
|
||||||
|
- `/os/1.0/system/resources` -- CPU, memory, hardware
|
||||||
|
- `/os/1.0/system/network` -- interfaces, DNS, routes
|
||||||
|
- `/os/1.0/system/update` -- channel, frequency, pending updates
|
||||||
|
- `/os/1.0/services/{ovn,iscsi,lvm,multipath,nvme,tailscale,usbip}`
|
||||||
|
|
||||||
|
## incusos-health helper
|
||||||
|
|
||||||
|
`incusos/helpers/incusos-health` queries the IncusOS API across all
|
||||||
|
cluster nodes. Actions:
|
||||||
|
|
||||||
|
| Flag | Purpose |
|
||||||
|
|------|---------|
|
||||||
|
| `--status` | Version, hostname, TPM status, Secure Boot |
|
||||||
|
| `--partitions` | Disk/partition layout, A/B state |
|
||||||
|
| `--tpm` | TPM details, Secure Boot certs, encryption keys |
|
||||||
|
| `--services` | Enabled/disabled services |
|
||||||
|
| `--network` | Interface config, DNS, gateway |
|
||||||
|
| `--update` | Channel, frequency, pending updates |
|
||||||
|
| `--all` | All of the above |
|
||||||
|
|
||||||
|
## Safety rules (MANDATORY)
|
||||||
|
|
||||||
|
- **NEVER hard-stop during first boot** -- corrupts TPM permanently,
|
||||||
|
requires full reinstallation. Hard-stop is safe ONLY for already-
|
||||||
|
provisioned nodes (TPM state already sealed).
|
||||||
|
- **One node at a time** for break-fix exercises. 3-node cluster needs
|
||||||
|
2/3 quorum. Taking down 2 nodes = cluster failure.
|
||||||
|
- **Proxmox snapshots** before every destructive test (VMID range 900-939).
|
||||||
|
- **Target node-03** (VMID 902) for destructive tests -- not the leader,
|
||||||
|
fewest workloads.
|
||||||
|
- **Verify health** with `incusos-health --all` before and after exercises.
|
||||||
|
- **Monitor via Grafana** during all exercises.
|
||||||
|
|
||||||
|
## Update mechanics
|
||||||
|
|
||||||
|
- `os_version` = current running version.
|
||||||
|
- `os_version_next` = version on standby partition (same as current if
|
||||||
|
no update pending, different if update downloaded).
|
||||||
|
- `needs_reboot` = true when update is downloaded and waiting for reboot.
|
||||||
|
- `auto_reboot: false` in lab -- updates download but don't reboot.
|
||||||
|
- Check frequency: 6 hours, stable channel.
|
||||||
|
- Nodes update independently (rolling, not coordinated).
|
||||||
|
|
||||||
|
## Break-fix exercises (defined, not yet executed)
|
||||||
|
|
||||||
|
1. **Normal Update Observation** -- watch download, reboot, A/B switch
|
||||||
|
2. **Simulated Failed Update** -- hard-stop node-03 mid-update, expect rollback
|
||||||
|
3. **Network Isolation** -- disconnect node-03 NIC, observe OVN tunnel loss
|
||||||
|
4. **Full Node Failure** -- hard-stop node-03, verify 2/3 quorum holds
|
||||||
|
|
||||||
|
See `notes/incusos-break-fix.md` for full exercise details, prerequisites,
|
||||||
|
expected behavior, and recovery steps.
|
||||||
|
|
@ -35,7 +35,8 @@ incus-contrib/
|
||||||
│ │ ├── proxmox-api # Authenticated API calls (handles ! in token)
|
│ │ ├── proxmox-api # Authenticated API calls (handles ! in token)
|
||||||
│ │ ├── aether-browser # Playwright browser automation for Aether web UI
|
│ │ ├── aether-browser # Playwright browser automation for Aether web UI
|
||||||
│ │ ├── ovn-inspect # OVN/OVS topology inspection (--nb/--sb/--ovs/--trace)
|
│ │ ├── ovn-inspect # OVN/OVS topology inspection (--nb/--sb/--ovs/--trace)
|
||||||
│ │ └── incus-mitm # API traffic capture and analysis (--capture/--live/--analyze)
|
│ │ ├── incus-mitm # API traffic capture and analysis (--capture/--live/--analyze)
|
||||||
|
│ │ └── incusos-health # IncusOS health inspection (--status/--tpm/--partitions/--all)
|
||||||
│ ├── lab-test # Guided lab validation (12 test phases)
|
│ ├── lab-test # Guided lab validation (12 test phases)
|
||||||
│ ├── observe-deploy # Single-VM deploy with console screenshots
|
│ ├── observe-deploy # Single-VM deploy with console screenshots
|
||||||
│ ├── proxmox.yaml # Proxmox connection (gitignored)
|
│ ├── proxmox.yaml # Proxmox connection (gitignored)
|
||||||
|
|
@ -192,6 +193,7 @@ which files are being edited:
|
||||||
| `observability.md` | deploy-observability, observability dashboards/guide |
|
| `observability.md` | deploy-observability, observability dashboards/guide |
|
||||||
| `ovn-internals.md` | ovn-inspect, ovn-deep-dive |
|
| `ovn-internals.md` | ovn-inspect, ovn-deep-dive |
|
||||||
| `aether-api-payloads.md` | incus-mitm, api-interception |
|
| `aether-api-payloads.md` | incus-mitm, api-interception |
|
||||||
|
| `incusos-immutability.md` | incusos-health, incusos-break-fix |
|
||||||
| `proxmox-ssh-rules.md` | **Always loaded** (no paths filter) |
|
| `proxmox-ssh-rules.md` | **Always loaded** (no paths filter) |
|
||||||
| `lab-infrastructure.md` | incusos-proxmox, lab-test, examples |
|
| `lab-infrastructure.md` | incusos-proxmox, lab-test, examples |
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,758 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
# incusos-health -- Inspect IncusOS node health via the /os/1.0 REST API
|
||||||
|
#
|
||||||
|
# Queries one or more IncusOS cluster nodes for status, storage, security,
|
||||||
|
# services, network, and update information.
|
||||||
|
#
|
||||||
|
# Usage: incusos-health [OPTIONS] [ACTION]
|
||||||
|
# Run 'incusos-health --help' for full usage information.
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
readonly VERSION="0.1.0"
|
||||||
|
readonly SCRIPT_NAME="incusos-health"
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Defaults
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
REMOTE="oc-node-01"
|
||||||
|
NODE=""
|
||||||
|
JSON_OUTPUT=false
|
||||||
|
DRY_RUN=false
|
||||||
|
VERBOSE=false
|
||||||
|
QUIET=false
|
||||||
|
ACTION=""
|
||||||
|
|
||||||
|
# Services to check (IncusOS built-in services)
|
||||||
|
readonly SERVICES=(iscsi lvm multipath nvme ovn tailscale usbip)
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Color and formatting
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
setup_colors() {
|
||||||
|
if [[ -t 1 ]] && [[ "${NO_COLOR:-}" != "1" ]] && [[ "${TERM:-}" != "dumb" ]]; then
|
||||||
|
RED='\033[0;31m'
|
||||||
|
GREEN='\033[0;32m'
|
||||||
|
YELLOW='\033[0;33m'
|
||||||
|
BLUE='\033[0;34m'
|
||||||
|
CYAN='\033[0;36m'
|
||||||
|
BOLD='\033[1m'
|
||||||
|
DIM='\033[2m'
|
||||||
|
RESET='\033[0m'
|
||||||
|
else
|
||||||
|
RED='' GREEN='' YELLOW='' BLUE='' CYAN='' BOLD='' DIM='' RESET=''
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
info() { [[ "$QUIET" == true ]] && return; echo -e "${BLUE}[info]${RESET} $*"; }
|
||||||
|
success() { [[ "$QUIET" == true ]] && return; echo -e "${GREEN}[ok]${RESET} $*"; }
|
||||||
|
warn() { echo -e "${YELLOW}[warn]${RESET} $*" >&2; }
|
||||||
|
error() { echo -e "${RED}[error]${RESET} $*" >&2; }
|
||||||
|
step() { [[ "$QUIET" == true ]] && return; echo -e "${CYAN}[step]${RESET} ${BOLD}$*${RESET}"; }
|
||||||
|
detail() { [[ "$VERBOSE" != true ]] && return; echo -e "${DIM} $*${RESET}"; }
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Help
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
usage() {
|
||||||
|
cat <<EOF
|
||||||
|
${BOLD}${SCRIPT_NAME}${RESET} v${VERSION} - Inspect IncusOS node health
|
||||||
|
|
||||||
|
${BOLD}USAGE${RESET}
|
||||||
|
${SCRIPT_NAME} [OPTIONS] ACTION
|
||||||
|
|
||||||
|
${BOLD}ACTIONS${RESET}
|
||||||
|
-s, --status Quick status overview (hostname, version, uptime,
|
||||||
|
memory, TPM, Secure Boot, trusted state, update)
|
||||||
|
-p, --partitions Storage and partition info (drives, ZFS pools,
|
||||||
|
volumes, encryption status)
|
||||||
|
-t, --tpm TPM and security details (TPM status, Secure Boot,
|
||||||
|
certificates, recovery keys, trusted state)
|
||||||
|
-S, --services Service configuration (iscsi, lvm, multipath, nvme,
|
||||||
|
ovn, tailscale, usbip; OVN details)
|
||||||
|
-N, --network Network configuration (interfaces, addresses, routes,
|
||||||
|
state, DNS)
|
||||||
|
-u, --update Update configuration and state (channel, frequency,
|
||||||
|
auto reboot, versions, last check)
|
||||||
|
-a, --all Run all of the above
|
||||||
|
|
||||||
|
${BOLD}OPTIONS${RESET}
|
||||||
|
-r, --remote NAME Incus remote (default: ${REMOTE})
|
||||||
|
--node NAME Specific node (default: all nodes in cluster)
|
||||||
|
-j, --json Output raw JSON instead of formatted text
|
||||||
|
-n, --dry-run Show what would be queried without executing
|
||||||
|
-v, --verbose Show detailed output
|
||||||
|
-q, --quiet Suppress informational output
|
||||||
|
-h, --help Show this help message
|
||||||
|
|
||||||
|
${BOLD}API ENDPOINTS${RESET}
|
||||||
|
/os/1.0 Environment (hostname, version, uptime)
|
||||||
|
/os/1.0/system/security TPM, Secure Boot, encryption
|
||||||
|
/os/1.0/system/storage Drives, ZFS pools, volumes
|
||||||
|
/os/1.0/system/resources CPU, memory
|
||||||
|
/os/1.0/system/network Interfaces, DNS
|
||||||
|
/os/1.0/system/update Update config and state
|
||||||
|
/os/1.0/services/SERVICE Per-service configuration
|
||||||
|
|
||||||
|
${BOLD}EXAMPLES${RESET}
|
||||||
|
# Quick status of all cluster nodes
|
||||||
|
${SCRIPT_NAME} --status
|
||||||
|
|
||||||
|
# Storage info for a specific node
|
||||||
|
${SCRIPT_NAME} --partitions --node oc-node-02
|
||||||
|
|
||||||
|
# All info as JSON
|
||||||
|
${SCRIPT_NAME} --all --json
|
||||||
|
|
||||||
|
# Different remote
|
||||||
|
${SCRIPT_NAME} --status --remote lab-node-01
|
||||||
|
|
||||||
|
# Preview queries without running
|
||||||
|
${SCRIPT_NAME} --all --dry-run
|
||||||
|
EOF
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Argument parsing
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
parse_args() {
|
||||||
|
while [[ $# -gt 0 ]]; do
|
||||||
|
case "$1" in
|
||||||
|
-s|--status) ACTION="status";;
|
||||||
|
-p|--partitions) ACTION="partitions";;
|
||||||
|
-t|--tpm) ACTION="tpm";;
|
||||||
|
-S|--services) ACTION="services";;
|
||||||
|
-N|--network) ACTION="network";;
|
||||||
|
-u|--update) ACTION="update";;
|
||||||
|
-a|--all) ACTION="all";;
|
||||||
|
-r|--remote) REMOTE="$2"; shift;;
|
||||||
|
--node) NODE="$2"; shift;;
|
||||||
|
-j|--json) JSON_OUTPUT=true;;
|
||||||
|
-n|--dry-run) DRY_RUN=true;;
|
||||||
|
-v|--verbose) VERBOSE=true;;
|
||||||
|
-q|--quiet) QUIET=true;;
|
||||||
|
-h|--help) usage; exit 0;;
|
||||||
|
*)
|
||||||
|
error "Unknown option: $1"
|
||||||
|
echo "Run '${SCRIPT_NAME} --help' for usage." >&2
|
||||||
|
exit 1
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
shift
|
||||||
|
done
|
||||||
|
|
||||||
|
if [[ -z "$ACTION" ]]; then
|
||||||
|
error "No action specified"
|
||||||
|
echo "Run '${SCRIPT_NAME} --help' for usage." >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Node discovery
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
discover_nodes() {
|
||||||
|
if [[ -n "$NODE" ]]; then
|
||||||
|
echo "$NODE"
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
|
||||||
|
local members
|
||||||
|
members=$(incus cluster list "${REMOTE}:" -f csv 2>/dev/null \
|
||||||
|
| cut -d',' -f1 \
|
||||||
|
| sed 's/^ *//;s/ *$//') || true
|
||||||
|
|
||||||
|
if [[ -z "$members" ]]; then
|
||||||
|
# Not a cluster or cannot list -- fall back to the remote itself
|
||||||
|
echo "$REMOTE"
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "$members"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Query helper
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
query_node() {
|
||||||
|
local node="$1" endpoint="$2"
|
||||||
|
|
||||||
|
if [[ "$DRY_RUN" == true ]]; then
|
||||||
|
info "(dry-run) incus query ${node}:${endpoint}"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
detail "incus query ${node}:${endpoint}"
|
||||||
|
incus query "${node}:${endpoint}" 2>/dev/null || echo "{}"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Section: status
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
show_status() {
|
||||||
|
local node="$1"
|
||||||
|
|
||||||
|
local env_json resources_json security_json update_json
|
||||||
|
env_json=$(query_node "$node" "/os/1.0")
|
||||||
|
resources_json=$(query_node "$node" "/os/1.0/system/resources")
|
||||||
|
security_json=$(query_node "$node" "/os/1.0/system/security")
|
||||||
|
update_json=$(query_node "$node" "/os/1.0/system/update")
|
||||||
|
|
||||||
|
if [[ "$DRY_RUN" == true ]]; then return; fi
|
||||||
|
|
||||||
|
if [[ "$JSON_OUTPUT" == true ]]; then
|
||||||
|
python3 -c "
|
||||||
|
import json, sys
|
||||||
|
print(json.dumps({
|
||||||
|
'environment': json.loads(sys.argv[1]),
|
||||||
|
'resources': json.loads(sys.argv[2]),
|
||||||
|
'security': json.loads(sys.argv[3]),
|
||||||
|
'update': json.loads(sys.argv[4]),
|
||||||
|
}, indent=2))
|
||||||
|
" "$env_json" "$resources_json" "$security_json" "$update_json"
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
|
||||||
|
python3 -c "
|
||||||
|
import json, sys
|
||||||
|
|
||||||
|
env = json.loads(sys.argv[1]).get('environment', json.loads(sys.argv[1]))
|
||||||
|
res = json.loads(sys.argv[2])
|
||||||
|
sec_raw = json.loads(sys.argv[3])
|
||||||
|
sec = sec_raw.get('state', sec_raw)
|
||||||
|
upd = json.loads(sys.argv[4])
|
||||||
|
|
||||||
|
hostname = env.get('hostname', 'unknown')
|
||||||
|
version = env.get('os_version', env.get('version', 'unknown'))
|
||||||
|
uptime_s = env.get('uptime', 0)
|
||||||
|
|
||||||
|
# Format uptime
|
||||||
|
days = uptime_s // 86400
|
||||||
|
hours = (uptime_s % 86400) // 3600
|
||||||
|
mins = (uptime_s % 3600) // 60
|
||||||
|
parts = []
|
||||||
|
if days: parts.append(f'{days}d')
|
||||||
|
if hours: parts.append(f'{hours}h')
|
||||||
|
parts.append(f'{mins}m')
|
||||||
|
uptime_str = ' '.join(parts)
|
||||||
|
|
||||||
|
# Memory
|
||||||
|
mem = res.get('memory', {})
|
||||||
|
mem_used = mem.get('used', 0)
|
||||||
|
mem_total = mem.get('total', 0)
|
||||||
|
|
||||||
|
def fmt_bytes(b):
|
||||||
|
for unit in ['B','KiB','MiB','GiB','TiB']:
|
||||||
|
if b < 1024:
|
||||||
|
return f'{b:.1f} {unit}'
|
||||||
|
b /= 1024
|
||||||
|
return f'{b:.1f} PiB'
|
||||||
|
|
||||||
|
mem_pct = (mem_used / mem_total * 100) if mem_total else 0
|
||||||
|
|
||||||
|
# Security (state level)
|
||||||
|
tpm_status = sec.get('tpm_status', 'unknown')
|
||||||
|
secboot_status = 'enabled' if sec.get('secure_boot_enabled', False) else 'disabled'
|
||||||
|
trusted = sec.get('system_state_is_trusted', 'unknown')
|
||||||
|
|
||||||
|
# Update
|
||||||
|
needs_reboot = upd.get('state', {}).get('needs_reboot', False)
|
||||||
|
last_check = upd.get('state', {}).get('last_check', 'never')
|
||||||
|
|
||||||
|
print(f' Hostname: {hostname}')
|
||||||
|
print(f' Version: {version}')
|
||||||
|
print(f' Uptime: {uptime_str}')
|
||||||
|
print(f' Memory: {fmt_bytes(mem_used)} / {fmt_bytes(mem_total)} ({mem_pct:.0f}%)')
|
||||||
|
print(f' TPM: {tpm_status}')
|
||||||
|
print(f' Secure Boot: {secboot_status}')
|
||||||
|
print(f' System trusted: {trusted}')
|
||||||
|
print(f' Needs reboot: {\"yes\" if needs_reboot else \"no\"}')
|
||||||
|
print(f' Last update: {last_check}')
|
||||||
|
" "$env_json" "$resources_json" "$security_json" "$update_json"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Section: partitions
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
show_partitions() {
|
||||||
|
local node="$1"
|
||||||
|
|
||||||
|
local storage_json security_json
|
||||||
|
storage_json=$(query_node "$node" "/os/1.0/system/storage")
|
||||||
|
security_json=$(query_node "$node" "/os/1.0/system/security")
|
||||||
|
|
||||||
|
if [[ "$DRY_RUN" == true ]]; then return; fi
|
||||||
|
|
||||||
|
if [[ "$JSON_OUTPUT" == true ]]; then
|
||||||
|
python3 -c "
|
||||||
|
import json, sys
|
||||||
|
print(json.dumps({
|
||||||
|
'storage': json.loads(sys.argv[1]),
|
||||||
|
'security': json.loads(sys.argv[2]),
|
||||||
|
}, indent=2))
|
||||||
|
" "$storage_json" "$security_json"
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
|
||||||
|
python3 -c "
|
||||||
|
import json, sys
|
||||||
|
|
||||||
|
stor_raw = json.loads(sys.argv[1])
|
||||||
|
sec_raw = json.loads(sys.argv[2])
|
||||||
|
stor = stor_raw.get('state', stor_raw)
|
||||||
|
sec = sec_raw.get('state', sec_raw)
|
||||||
|
|
||||||
|
def fmt_bytes(b):
|
||||||
|
for unit in ['B','KiB','MiB','GiB','TiB']:
|
||||||
|
if b < 1024:
|
||||||
|
return f'{b:.1f} {unit}'
|
||||||
|
b /= 1024
|
||||||
|
return f'{b:.1f} PiB'
|
||||||
|
|
||||||
|
# Drives
|
||||||
|
disks = stor.get('drives', stor.get('disks', []))
|
||||||
|
if disks:
|
||||||
|
print(' Drives:')
|
||||||
|
for d in disks:
|
||||||
|
model = d.get('model_name', d.get('model', 'unknown'))
|
||||||
|
bus = d.get('bus', 'unknown')
|
||||||
|
size = d.get('capacity_in_bytes', d.get('size', 0))
|
||||||
|
boot = ' [boot]' if d.get('boot') else ''
|
||||||
|
did = d.get('id', '?').split('/')[-1]
|
||||||
|
pool = d.get('member_pool', '')
|
||||||
|
pool_str = f' pool={pool}' if pool else ''
|
||||||
|
print(f' {did}: {model} ({bus}, {fmt_bytes(size)}){boot}{pool_str}')
|
||||||
|
else:
|
||||||
|
print(' Drives: none found')
|
||||||
|
|
||||||
|
# ZFS pools
|
||||||
|
pools = stor.get('pools', [])
|
||||||
|
if pools:
|
||||||
|
print(' ZFS pools:')
|
||||||
|
for p in pools:
|
||||||
|
name = p.get('name', 'unknown')
|
||||||
|
state = p.get('state', 'unknown')
|
||||||
|
ptype = p.get('type', 'unknown')
|
||||||
|
alloc = p.get('pool_allocated_space_in_bytes', 0)
|
||||||
|
total = p.get('raw_pool_size_in_bytes', p.get('usable_pool_size_in_bytes', 0))
|
||||||
|
pct = (alloc / total * 100) if total else 0
|
||||||
|
enc_key = p.get('encryption_key_status', 'N/A')
|
||||||
|
print(f' {name}: {state} ({ptype}, {fmt_bytes(alloc)} / {fmt_bytes(total)}, {pct:.0f}%) enc_key={enc_key}')
|
||||||
|
for v in p.get('volumes', []):
|
||||||
|
vname = v.get('name', '?')
|
||||||
|
vuse = v.get('use', '?')
|
||||||
|
vusage = v.get('usage_in_bytes', 0)
|
||||||
|
vquota = v.get('quota_in_bytes', 0)
|
||||||
|
quota_str = fmt_bytes(vquota) if vquota else 'no quota'
|
||||||
|
print(f' vol {vname} ({vuse}): {fmt_bytes(vusage)} / {quota_str}')
|
||||||
|
else:
|
||||||
|
print(' ZFS pools: none found')
|
||||||
|
|
||||||
|
# Encryption
|
||||||
|
encrypted_vols = sec.get('encrypted_volumes', [])
|
||||||
|
if encrypted_vols:
|
||||||
|
print(' Encryption:')
|
||||||
|
for v in encrypted_vols:
|
||||||
|
vol = v.get('volume', '?')
|
||||||
|
state = v.get('state', '?')
|
||||||
|
print(f' {vol}: {state}')
|
||||||
|
else:
|
||||||
|
print(' Encryption: not configured')
|
||||||
|
" "$storage_json" "$security_json"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Section: tpm
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
show_tpm() {
|
||||||
|
local node="$1"
|
||||||
|
|
||||||
|
local security_json
|
||||||
|
security_json=$(query_node "$node" "/os/1.0/system/security")
|
||||||
|
|
||||||
|
if [[ "$DRY_RUN" == true ]]; then return; fi
|
||||||
|
|
||||||
|
if [[ "$JSON_OUTPUT" == true ]]; then
|
||||||
|
python3 -c "
|
||||||
|
import json, sys
|
||||||
|
print(json.dumps(json.loads(sys.argv[1]), indent=2))
|
||||||
|
" "$security_json"
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
|
||||||
|
python3 -c "
|
||||||
|
import json, sys
|
||||||
|
|
||||||
|
sec_raw = json.loads(sys.argv[1])
|
||||||
|
sec = sec_raw.get('state', sec_raw)
|
||||||
|
sec_cfg = sec_raw.get('config', {})
|
||||||
|
|
||||||
|
# TPM
|
||||||
|
tpm_status = sec.get('tpm_status', 'unknown')
|
||||||
|
print(f' TPM status: {tpm_status}')
|
||||||
|
|
||||||
|
# Secure Boot
|
||||||
|
sb_enabled = sec.get('secure_boot_enabled', False)
|
||||||
|
print(f' Secure Boot: {\"enabled\" if sb_enabled else \"disabled\"}')
|
||||||
|
|
||||||
|
# Certificates
|
||||||
|
certs = sec.get('secure_boot_certificates', [])
|
||||||
|
if certs:
|
||||||
|
print(' Certificates:')
|
||||||
|
for c in certs:
|
||||||
|
ctype = c.get('type', '?')
|
||||||
|
subject = c.get('subject', 'unknown')
|
||||||
|
fp = c.get('fingerprint', '')
|
||||||
|
fp_short = fp[:16] + '...' if len(fp) > 16 else fp
|
||||||
|
print(f' {ctype:4s} {subject}')
|
||||||
|
if fp_short:
|
||||||
|
print(f' fingerprint: {fp_short}')
|
||||||
|
|
||||||
|
# Encrypted volumes
|
||||||
|
enc_vols = sec.get('encrypted_volumes', [])
|
||||||
|
if enc_vols:
|
||||||
|
print(' Encrypted volumes:')
|
||||||
|
for v in enc_vols:
|
||||||
|
print(f' {v.get(\"volume\", \"?\")}: {v.get(\"state\", \"?\")}')
|
||||||
|
|
||||||
|
# Recovery keys
|
||||||
|
recovery_keys = sec_cfg.get('encryption_recovery_keys', [])
|
||||||
|
retrieved = sec.get('encryption_recovery_keys_retrieved', False)
|
||||||
|
print(f' Recovery keys: {len(recovery_keys)} key(s), retrieved={retrieved}')
|
||||||
|
|
||||||
|
# Pool recovery keys
|
||||||
|
pool_keys = sec.get('pool_recovery_keys', {})
|
||||||
|
if pool_keys:
|
||||||
|
print(f' Pool keys: {list(pool_keys.keys())}')
|
||||||
|
|
||||||
|
# System trusted
|
||||||
|
trusted = sec.get('system_state_is_trusted', 'unknown')
|
||||||
|
print(f' System trusted: {trusted}')
|
||||||
|
" "$security_json"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Section: services
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
show_services() {
|
||||||
|
local node="$1"
|
||||||
|
|
||||||
|
if [[ "$JSON_OUTPUT" == true ]]; then
|
||||||
|
local all_json="{"
|
||||||
|
local first=true
|
||||||
|
for svc in "${SERVICES[@]}"; do
|
||||||
|
local svc_json
|
||||||
|
svc_json=$(query_node "$node" "/os/1.0/services/${svc}")
|
||||||
|
if [[ "$DRY_RUN" == true ]]; then continue; fi
|
||||||
|
if [[ "$first" == true ]]; then
|
||||||
|
first=false
|
||||||
|
else
|
||||||
|
all_json+=","
|
||||||
|
fi
|
||||||
|
all_json+="\"${svc}\":${svc_json}"
|
||||||
|
done
|
||||||
|
all_json+="}"
|
||||||
|
if [[ "$DRY_RUN" != true ]]; then
|
||||||
|
echo "$all_json" | python3 -m json.tool
|
||||||
|
fi
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
|
||||||
|
for svc in "${SERVICES[@]}"; do
|
||||||
|
local svc_json
|
||||||
|
svc_json=$(query_node "$node" "/os/1.0/services/${svc}")
|
||||||
|
if [[ "$DRY_RUN" == true ]]; then continue; fi
|
||||||
|
|
||||||
|
python3 -c "
|
||||||
|
import json, sys
|
||||||
|
|
||||||
|
svc_name = sys.argv[1]
|
||||||
|
data = json.loads(sys.argv[2])
|
||||||
|
|
||||||
|
config = data.get('config', data)
|
||||||
|
enabled = config.get('enabled', False)
|
||||||
|
status = 'enabled' if enabled else 'disabled'
|
||||||
|
|
||||||
|
line = f' {svc_name:12s} {status}'
|
||||||
|
print(line)
|
||||||
|
|
||||||
|
# Extra details for OVN
|
||||||
|
if svc_name == 'ovn' and enabled:
|
||||||
|
db = config.get('database', '')
|
||||||
|
tunnel_addr = config.get('tunnel_address', '')
|
||||||
|
tunnel_proto = config.get('tunnel_protocol', '')
|
||||||
|
if db:
|
||||||
|
print(f' database: {db}')
|
||||||
|
if tunnel_addr:
|
||||||
|
print(f' tunnel address: {tunnel_addr}')
|
||||||
|
if tunnel_proto:
|
||||||
|
print(f' tunnel protocol: {tunnel_proto}')
|
||||||
|
" "$svc" "$svc_json"
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Section: network
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
show_network() {
|
||||||
|
local node="$1"
|
||||||
|
|
||||||
|
local network_json
|
||||||
|
network_json=$(query_node "$node" "/os/1.0/system/network")
|
||||||
|
|
||||||
|
if [[ "$DRY_RUN" == true ]]; then return; fi
|
||||||
|
|
||||||
|
if [[ "$JSON_OUTPUT" == true ]]; then
|
||||||
|
echo "$network_json" | python3 -m json.tool
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
|
||||||
|
python3 -c "
|
||||||
|
import json, sys
|
||||||
|
|
||||||
|
net = json.loads(sys.argv[1])
|
||||||
|
|
||||||
|
def fmt_bytes(b):
|
||||||
|
for unit in ['B','KiB','MiB','GiB','TiB']:
|
||||||
|
if b < 1024: return f'{b:.1f} {unit}'
|
||||||
|
b /= 1024
|
||||||
|
return f'{b:.1f} PiB'
|
||||||
|
|
||||||
|
# Interfaces
|
||||||
|
ifaces = net.get('interfaces', net.get('config', {}).get('interfaces', []))
|
||||||
|
if not isinstance(ifaces, list): ifaces = []
|
||||||
|
print(' Interfaces:')
|
||||||
|
for iface in ifaces:
|
||||||
|
name = iface.get('name', 'unknown')
|
||||||
|
hwaddr = iface.get('hwaddr', iface.get('mac', ''))
|
||||||
|
roles = iface.get('roles', [])
|
||||||
|
roles_str = ', '.join(roles) if isinstance(roles, list) and roles else ''
|
||||||
|
addrs = iface.get('addresses', [])
|
||||||
|
addr_strs = [a if isinstance(a, str) else a.get('address', '?') for a in addrs]
|
||||||
|
state = iface.get('state', {})
|
||||||
|
|
||||||
|
header = f' {name}'
|
||||||
|
if hwaddr: header += f' ({hwaddr})'
|
||||||
|
if roles_str: header += f' [{roles_str}]'
|
||||||
|
print(header)
|
||||||
|
|
||||||
|
if addr_strs:
|
||||||
|
print(f' addresses: {\", \".join(addr_strs)}')
|
||||||
|
|
||||||
|
link_state = state.get('state', state.get('link', ''))
|
||||||
|
mtu = state.get('mtu', iface.get('mtu', ''))
|
||||||
|
speed = state.get('speed', '')
|
||||||
|
if link_state:
|
||||||
|
parts = [f'state={link_state}']
|
||||||
|
if mtu: parts.append(f'mtu={mtu}')
|
||||||
|
if speed: parts.append(f'speed={speed}')
|
||||||
|
print(f' link: {\", \".join(parts)}')
|
||||||
|
|
||||||
|
routes = iface.get('routes', [])
|
||||||
|
if isinstance(routes, list) and routes:
|
||||||
|
route_strs = []
|
||||||
|
for r in routes:
|
||||||
|
if isinstance(r, str): route_strs.append(r)
|
||||||
|
elif isinstance(r, dict):
|
||||||
|
s = r.get('destination', r.get('to', '?'))
|
||||||
|
via = r.get('via', r.get('gateway', ''))
|
||||||
|
if via: s += f' via {via}'
|
||||||
|
route_strs.append(s)
|
||||||
|
if route_strs:
|
||||||
|
print(f' routes: {route_strs[0]}')
|
||||||
|
for rs in route_strs[1:]:
|
||||||
|
print(f' {rs}')
|
||||||
|
|
||||||
|
stats = state.get('stats', iface.get('stats', {}))
|
||||||
|
if isinstance(stats, dict) and stats:
|
||||||
|
rx = stats.get('rx_bytes', stats.get('rx', 0))
|
||||||
|
tx = stats.get('tx_bytes', stats.get('tx', 0))
|
||||||
|
if rx or tx:
|
||||||
|
print(f' stats: rx={fmt_bytes(rx)}, tx={fmt_bytes(tx)}')
|
||||||
|
|
||||||
|
# DNS
|
||||||
|
dns = net.get('dns', net.get('config', {}).get('dns', {}))
|
||||||
|
if isinstance(dns, dict) and dns:
|
||||||
|
print(' DNS:')
|
||||||
|
hostname = dns.get('hostname', dns.get('search', ''))
|
||||||
|
nameservers = dns.get('nameservers', dns.get('servers', []))
|
||||||
|
if hostname: print(f' hostname: {hostname}')
|
||||||
|
if isinstance(nameservers, list) and nameservers:
|
||||||
|
print(f' nameservers: {\", \".join(str(n) for n in nameservers)}')
|
||||||
|
" "$network_json"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Section: update
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
show_update() {
|
||||||
|
local node="$1"
|
||||||
|
|
||||||
|
local update_json env_json
|
||||||
|
update_json=$(query_node "$node" "/os/1.0/system/update")
|
||||||
|
env_json=$(query_node "$node" "/os/1.0")
|
||||||
|
|
||||||
|
if [[ "$DRY_RUN" == true ]]; then return; fi
|
||||||
|
|
||||||
|
if [[ "$JSON_OUTPUT" == true ]]; then
|
||||||
|
echo "$update_json" | python3 -m json.tool
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
|
||||||
|
python3 -c "
|
||||||
|
import json, sys
|
||||||
|
|
||||||
|
upd = json.loads(sys.argv[1])
|
||||||
|
env = json.loads(sys.argv[2]).get('environment', json.loads(sys.argv[2]))
|
||||||
|
config = upd.get('config', {})
|
||||||
|
state = upd.get('state', {})
|
||||||
|
|
||||||
|
channel = config.get('channel', 'unknown')
|
||||||
|
frequency = config.get('check_frequency', 'unknown')
|
||||||
|
auto_reboot = config.get('auto_reboot', 'unknown')
|
||||||
|
|
||||||
|
current = env.get('os_version', 'unknown')
|
||||||
|
next_ver = env.get('os_version_next', 'none')
|
||||||
|
update_status = state.get('status', '')
|
||||||
|
last_check = state.get('last_check', 'never')
|
||||||
|
needs_reboot = state.get('needs_reboot', False)
|
||||||
|
|
||||||
|
print(f' Channel: {channel}')
|
||||||
|
print(f' Check freq: {frequency}')
|
||||||
|
print(f' Auto reboot: {auto_reboot}')
|
||||||
|
print(f' Current: {current}')
|
||||||
|
print(f' Next version: {next_ver}')
|
||||||
|
print(f' Update status: {update_status}')
|
||||||
|
print(f' Last check: {last_check}')
|
||||||
|
print(f' Needs reboot: {\"yes\" if needs_reboot else \"no\"}')
|
||||||
|
" "$update_json" "$env_json"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Run a section across all nodes
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
run_section() {
|
||||||
|
local section_name="$1"
|
||||||
|
local section_fn="$2"
|
||||||
|
shift 2
|
||||||
|
local nodes=("$@")
|
||||||
|
|
||||||
|
step "$section_name"
|
||||||
|
|
||||||
|
for node in "${nodes[@]}"; do
|
||||||
|
echo -e " ${BOLD}--- ${node} ---${RESET}"
|
||||||
|
"$section_fn" "$node"
|
||||||
|
echo
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
run_section_json() {
|
||||||
|
local section_fn="$1"
|
||||||
|
shift
|
||||||
|
local nodes=("$@")
|
||||||
|
|
||||||
|
if [[ ${#nodes[@]} -eq 1 ]]; then
|
||||||
|
"$section_fn" "${nodes[0]}"
|
||||||
|
else
|
||||||
|
echo "{"
|
||||||
|
local first=true
|
||||||
|
for node in "${nodes[@]}"; do
|
||||||
|
if [[ "$first" == true ]]; then
|
||||||
|
first=false
|
||||||
|
else
|
||||||
|
echo ","
|
||||||
|
fi
|
||||||
|
echo " \"${node}\":"
|
||||||
|
"$section_fn" "$node" | sed 's/^/ /'
|
||||||
|
done
|
||||||
|
echo "}"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Main
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
main() {
|
||||||
|
setup_colors
|
||||||
|
parse_args "$@"
|
||||||
|
|
||||||
|
# Discover cluster nodes
|
||||||
|
if [[ "$DRY_RUN" == true ]]; then
|
||||||
|
if [[ -n "$NODE" ]]; then
|
||||||
|
info "(dry-run) Would query node: ${NODE}"
|
||||||
|
else
|
||||||
|
info "(dry-run) Would discover nodes via: incus cluster list ${REMOTE}: -f csv"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
local nodes_raw
|
||||||
|
nodes_raw=$(discover_nodes)
|
||||||
|
|
||||||
|
local nodes=()
|
||||||
|
while IFS= read -r line; do
|
||||||
|
[[ -n "$line" ]] && nodes+=("$line")
|
||||||
|
done <<< "$nodes_raw"
|
||||||
|
|
||||||
|
if [[ ${#nodes[@]} -eq 0 ]]; then
|
||||||
|
error "No nodes found. Check that remote '${REMOTE}' is accessible."
|
||||||
|
error " incus remote list"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ "$DRY_RUN" != true ]]; then
|
||||||
|
info "Nodes: ${nodes[*]}"
|
||||||
|
fi
|
||||||
|
echo
|
||||||
|
|
||||||
|
# Section lookup: action -> (title, function)
|
||||||
|
local -A SECTION_TITLE=(
|
||||||
|
[status]="Status" [partitions]="Storage & Partitions"
|
||||||
|
[tpm]="TPM & Security" [services]="Services"
|
||||||
|
[network]="Network" [update]="Update"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Build section list
|
||||||
|
local sections=()
|
||||||
|
if [[ "$ACTION" == "all" ]]; then
|
||||||
|
sections=(status partitions tpm services network update)
|
||||||
|
else
|
||||||
|
sections=("$ACTION")
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Dispatch
|
||||||
|
if [[ "$JSON_OUTPUT" == true && ${#sections[@]} -gt 1 ]]; then
|
||||||
|
echo "{"
|
||||||
|
local i=0
|
||||||
|
for section in "${sections[@]}"; do
|
||||||
|
echo " \"${section}\":"
|
||||||
|
run_section_json "show_${section}" "${nodes[@]}" | sed 's/^/ /'
|
||||||
|
i=$((i + 1))
|
||||||
|
[[ $i -lt ${#sections[@]} ]] && echo ","
|
||||||
|
done
|
||||||
|
echo "}"
|
||||||
|
else
|
||||||
|
for section in "${sections[@]}"; do
|
||||||
|
if [[ "$JSON_OUTPUT" == true ]]; then
|
||||||
|
run_section_json "show_${section}" "${nodes[@]}"
|
||||||
|
else
|
||||||
|
run_section "${SECTION_TITLE[$section]}" "show_${section}" "${nodes[@]}"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
main "$@"
|
||||||
|
|
@ -0,0 +1,421 @@
|
||||||
|
# IncusOS Break-Fix Lab -- Immutability Exploration & Resilience Testing
|
||||||
|
|
||||||
|
IncusOS is an immutable, purpose-built operating system for running Incus
|
||||||
|
clusters. This guide documents what we discovered about its internals through
|
||||||
|
the `/os/1.0` API, and defines a catalog of break-fix exercises for testing
|
||||||
|
cluster resilience in a safe lab environment.
|
||||||
|
|
||||||
|
All observations come from a 3-node Proxmox-hosted cluster (VMID 900-902)
|
||||||
|
running IncusOS with virtual TPM, Secure Boot, and OVN networking.
|
||||||
|
|
||||||
|
## IncusOS Architecture
|
||||||
|
|
||||||
|
IncusOS is an immutable OS designed for a single purpose: running Incus.
|
||||||
|
Key architectural properties discovered via the `/os/1.0` API:
|
||||||
|
|
||||||
|
- **Immutable root filesystem** with A/B partition scheme. The running system
|
||||||
|
boots from one partition; updates install to the other. Rollback is automatic
|
||||||
|
if the new partition fails validation.
|
||||||
|
- **TPM-based full disk encryption** -- root and swap volumes are encrypted
|
||||||
|
and unlocked automatically via TPM measured boot. No passphrase required
|
||||||
|
during normal operation.
|
||||||
|
- **Secure Boot enforced** with 4 certificates in the UEFI firmware:
|
||||||
|
- PK (Platform Key) -- root of trust
|
||||||
|
- KEK (Key Exchange Key) -- authorizes db updates
|
||||||
|
- db (2025 signing cert) -- validates current signed binaries
|
||||||
|
- db (2026 signing cert) -- validates next-year signed binaries
|
||||||
|
- **Version format**: `YYYYMMDDHHMI` (build timestamp, not semver).
|
||||||
|
Example: `202602240349` = 2026-02-24 at 03:49 UTC.
|
||||||
|
- **ZFS storage**: pool "local" on a dedicated partition (raid0 on single
|
||||||
|
disk, ~30 GiB usable), encrypted with its own pool recovery key.
|
||||||
|
- **Update system**: stable channel, 6-hour check frequency,
|
||||||
|
`auto_reboot: false` (updates download but do not reboot automatically).
|
||||||
|
|
||||||
|
### Lab cluster state
|
||||||
|
|
||||||
|
| Node | VMID | IP | IncusOS Version |
|
||||||
|
|------|------|----|-----------------|
|
||||||
|
| node-01 | 900 | 192.168.102.140 | 202602240349 |
|
||||||
|
| node-02 | 901 | 192.168.102.141 | 202602230420 |
|
||||||
|
| node-03 | 902 | 192.168.102.142 | 202602230420 |
|
||||||
|
|
||||||
|
node-01 received a newer build than nodes 02/03, proving that nodes update
|
||||||
|
independently (rolling updates, not cluster-wide atomic upgrades).
|
||||||
|
|
||||||
|
## Partition & Disk Layout
|
||||||
|
|
||||||
|
Each node has a single 64 GiB SCSI disk (QEMU HARDDISK) with this layout:
|
||||||
|
|
||||||
|
```
|
||||||
|
+------------------------------------------------------+
|
||||||
|
| SCSI Disk (64 GiB, QEMU HARDDISK) |
|
||||||
|
|------------------------------------------------------|
|
||||||
|
| Partition 1 EFI System Partition |
|
||||||
|
| Partition A Root filesystem (active or standby) |
|
||||||
|
| Partition B Root filesystem (standby or active) |
|
||||||
|
| Partition X Swap (encrypted, TPM-unlocked) |
|
||||||
|
| Partition 11 ZFS data pool ("local") |
|
||||||
|
+------------------------------------------------------+
|
||||||
|
```
|
||||||
|
|
||||||
|
Key observations from the storage API:
|
||||||
|
|
||||||
|
- **Partition 11** is the ZFS data partition, hosting pool "local".
|
||||||
|
- **Pool "local"**: raid0, single vdev, ~30 GiB total, contains volume
|
||||||
|
"incus" (~10 GiB used for Incus database, images, instances).
|
||||||
|
- **Root + swap**: both encrypted, both unlocked by TPM at boot.
|
||||||
|
No manual key entry needed unless TPM state is corrupted.
|
||||||
|
|
||||||
|
## Security Chain
|
||||||
|
|
||||||
|
The boot security chain, as validated through the API:
|
||||||
|
|
||||||
|
```
|
||||||
|
UEFI firmware
|
||||||
|
--> Secure Boot certificate validation (PK -> KEK -> db certs)
|
||||||
|
--> Signed kernel + initrd loaded
|
||||||
|
--> TPM measured boot (PCR measurements recorded)
|
||||||
|
--> TPM validates measurements match expected policy
|
||||||
|
--> Root partition decrypted and mounted
|
||||||
|
--> Swap partition decrypted and activated
|
||||||
|
--> System boots into trusted state
|
||||||
|
```
|
||||||
|
|
||||||
|
API-reported security state:
|
||||||
|
|
||||||
|
| Property | Value |
|
||||||
|
|----------|-------|
|
||||||
|
| TPM status | ok |
|
||||||
|
| Secure Boot | enabled, enforced |
|
||||||
|
| System state | trusted |
|
||||||
|
| Encryption recovery keys | 1 key (retrievable via API) |
|
||||||
|
| Pool recovery keys | 1 key for pool "local" |
|
||||||
|
|
||||||
|
**Recovery keys** are a safety net. If TPM state is corrupted (e.g., by a
|
||||||
|
hard-stop during first boot), the encryption recovery key allows manual
|
||||||
|
unlock. The pool recovery key allows ZFS pool import on a different system.
|
||||||
|
|
||||||
|
## Services Configuration
|
||||||
|
|
||||||
|
Discovered via `/os/1.0/services/*`:
|
||||||
|
|
||||||
|
| Service | Status | Notes |
|
||||||
|
|---------|--------|-------|
|
||||||
|
| OVN | enabled | Connects to cluster DB at `tcp:192.168.102.142:6642`, Geneve tunnels |
|
||||||
|
| iSCSI | disabled | |
|
||||||
|
| LVM | disabled | |
|
||||||
|
| Multipath | disabled | |
|
||||||
|
| NVMe | disabled | |
|
||||||
|
| Tailscale | disabled | |
|
||||||
|
| USB/IP | disabled | |
|
||||||
|
|
||||||
|
OVN is the only enabled service beyond Incus itself, providing the overlay
|
||||||
|
network for cross-node container connectivity and network policies.
|
||||||
|
|
||||||
|
## Network Configuration
|
||||||
|
|
||||||
|
Each node has a single management interface:
|
||||||
|
|
||||||
|
| Property | Value |
|
||||||
|
|----------|-------|
|
||||||
|
| Interface name | mgmt |
|
||||||
|
| Role | management + cluster traffic |
|
||||||
|
| Addressing | Static |
|
||||||
|
| Subnet | /22 (192.168.100.0/22) |
|
||||||
|
| DNS | 192.168.100.1 |
|
||||||
|
| Gateway | 192.168.100.1 |
|
||||||
|
|
||||||
|
Node IP assignments:
|
||||||
|
- node-01: `192.168.102.140/22`
|
||||||
|
- node-02: `192.168.102.141/22`
|
||||||
|
- node-03: `192.168.102.142/22`
|
||||||
|
|
||||||
|
The management interface carries both management traffic (API, cluster
|
||||||
|
heartbeats) and OVN Geneve tunnel traffic. In production, these should
|
||||||
|
be separated.
|
||||||
|
|
||||||
|
## Update Mechanics (Observed)
|
||||||
|
|
||||||
|
The IncusOS update system uses A/B partitions for safe, rollback-capable
|
||||||
|
updates. Observations from querying the update API:
|
||||||
|
|
||||||
|
1. **No pending update**: `os_version` and `os_version_next` are identical.
|
||||||
|
This means no update has been downloaded or is waiting for reboot.
|
||||||
|
|
||||||
|
2. **Pending update**: `os_version_next` would differ from `os_version`,
|
||||||
|
indicating a new build has been downloaded to the standby partition.
|
||||||
|
|
||||||
|
3. **Auto-reboot disabled**: `auto_reboot: false` means updates download
|
||||||
|
to the standby partition but the node continues running the current
|
||||||
|
version until explicitly rebooted (or until an admin triggers reboot
|
||||||
|
via API / UI).
|
||||||
|
|
||||||
|
4. **No reboot needed**: `needs_reboot: false` confirms no downloaded
|
||||||
|
update is waiting for a reboot to activate.
|
||||||
|
|
||||||
|
5. **Independent node updates**: node-01 is on `202602240349` while
|
||||||
|
nodes 02/03 are on `202602230420`. This proves each node checks for
|
||||||
|
and applies updates independently. There is no cluster-wide coordinated
|
||||||
|
update mechanism at the OS level.
|
||||||
|
|
||||||
|
6. **Check frequency**: 6 hours. The stable channel is checked
|
||||||
|
automatically on this interval.
|
||||||
|
|
||||||
|
### Update lifecycle (theoretical)
|
||||||
|
|
||||||
|
```
|
||||||
|
Check timer fires (every 6h)
|
||||||
|
--> Query stable channel for new version
|
||||||
|
--> Download new rootfs to standby partition (A or B)
|
||||||
|
--> os_version_next updated, needs_reboot = true
|
||||||
|
--> (if auto_reboot) Reboot automatically
|
||||||
|
--> (if !auto_reboot) Wait for manual reboot
|
||||||
|
--> On reboot: boot from new partition
|
||||||
|
--> TPM re-measures, validates new boot chain
|
||||||
|
--> If valid: new partition becomes active
|
||||||
|
--> If invalid: rollback to previous partition
|
||||||
|
```
|
||||||
|
|
||||||
|
## Break-Fix Exercise Catalog
|
||||||
|
|
||||||
|
These exercises are designed to test IncusOS cluster resilience in the
|
||||||
|
Proxmox lab. All exercises follow strict safety rules (see Safety Rules
|
||||||
|
section below).
|
||||||
|
|
||||||
|
**Current status**: All exercises are defined but not yet executed.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Exercise 1: Normal Update Observation
|
||||||
|
|
||||||
|
**Goal**: Observe the full update lifecycle -- download, reboot, A/B
|
||||||
|
partition switch, version verification.
|
||||||
|
|
||||||
|
**Prerequisites**:
|
||||||
|
- Proxmox snapshot of all cluster nodes (VMID 900-902)
|
||||||
|
- Grafana monitoring active (observe cluster metrics during update)
|
||||||
|
- Verify cluster health with `incusos-health --all`
|
||||||
|
|
||||||
|
**Steps**:
|
||||||
|
1. Record current versions: `incusos-health --update`
|
||||||
|
2. Trigger update via Operations Center UI, or wait for 6h check interval
|
||||||
|
3. Monitor via Grafana for download activity and node state changes
|
||||||
|
4. When `needs_reboot: true`, reboot one node at a time
|
||||||
|
5. After reboot, verify new version: `incusos-health --status`
|
||||||
|
6. Confirm A/B partition switch: `incusos-health --partitions`
|
||||||
|
|
||||||
|
**What to observe**:
|
||||||
|
- Download phase duration
|
||||||
|
- Reboot duration (typically 30-60s for IncusOS)
|
||||||
|
- Cluster behavior while one node reboots (2/3 quorum maintained)
|
||||||
|
- New version number in `os_version`
|
||||||
|
- Previous version still available on standby partition
|
||||||
|
|
||||||
|
**Status**: Defined, not yet executed.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Exercise 2: Simulated Failed Update (Hard-Stop Mid-Update)
|
||||||
|
|
||||||
|
**Goal**: Verify that IncusOS rolls back to the previous partition after
|
||||||
|
a failed update (simulated by hard-stopping the VM during update).
|
||||||
|
|
||||||
|
**Prerequisites**:
|
||||||
|
- Proxmox snapshot of node-03 (VMID 902) -- non-leader, fewest workloads
|
||||||
|
- Verify node-03 has no critical workloads
|
||||||
|
- Confirm 3/3 nodes healthy before starting
|
||||||
|
|
||||||
|
**Target node**: node-03 (VMID 902) ONLY. Never the cluster leader.
|
||||||
|
|
||||||
|
**Steps**:
|
||||||
|
1. Take Proxmox snapshot of node-03: `incusos/helpers/proxmox-api POST /nodes/pve/qemu/902/snapshot -d '{"snapname":"pre-break-fix-2"}'`
|
||||||
|
2. Trigger an update on node-03 (if one is available)
|
||||||
|
3. During the update download or early reboot phase, hard-stop node-03
|
||||||
|
via Proxmox API: `incusos/helpers/proxmox-api POST /nodes/pve/qemu/902/status/stop`
|
||||||
|
4. Wait 10 seconds, then start node-03:
|
||||||
|
`incusos/helpers/proxmox-api POST /nodes/pve/qemu/902/status/start`
|
||||||
|
5. Monitor boot via console screenshots: `incusos/helpers/proxmox-screenshot 902`
|
||||||
|
6. After boot, check version: `incusos-health --status` on node-03
|
||||||
|
|
||||||
|
**Expected behavior**:
|
||||||
|
- Node boots from the previous (known-good) partition
|
||||||
|
- Failed partition is marked as bad / not bootable
|
||||||
|
- `os_version` returns to the pre-update version
|
||||||
|
- Node rejoins the cluster automatically
|
||||||
|
|
||||||
|
**Safety**:
|
||||||
|
- ONLY node-03 -- maintains 2/3 quorum (node-01 + node-02 continue)
|
||||||
|
- Do NOT hard-stop during *first boot* (corrupts TPM permanently)
|
||||||
|
- If recovery fails, restore from Proxmox snapshot
|
||||||
|
|
||||||
|
**Status**: Defined, not yet executed.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Exercise 3: Network Isolation
|
||||||
|
|
||||||
|
**Goal**: Observe how the cluster handles a node losing network
|
||||||
|
connectivity -- OVN tunnel loss, cluster membership changes, and
|
||||||
|
automatic recovery on reconnection.
|
||||||
|
|
||||||
|
**Prerequisites**:
|
||||||
|
- Evacuate all workloads from node-03 before disconnecting
|
||||||
|
- Grafana monitoring active (watch OVN tunnel metrics, cluster events)
|
||||||
|
- Verify cluster health: `incusos-health --all`
|
||||||
|
|
||||||
|
**Steps**:
|
||||||
|
1. Evacuate workloads from node-03:
|
||||||
|
`incus cluster evacuate oc-node-03 --target oc-node-01`
|
||||||
|
2. Disconnect node-03 NIC via Proxmox API:
|
||||||
|
`incusos/helpers/proxmox-api PUT /nodes/pve/qemu/902/config -d '{"net0":"virtio=...,link_down=1"}'`
|
||||||
|
3. Observe in Grafana: OVN Geneve tunnel loss, cluster detecting missing node
|
||||||
|
4. Wait 2-5 minutes for cluster to mark node-03 as offline
|
||||||
|
5. Reconnect NIC: set `link_down=0` via Proxmox API
|
||||||
|
6. Monitor recovery: node-03 should rejoin cluster, OVN tunnels re-establish
|
||||||
|
7. Measure total recovery time from reconnection to healthy state
|
||||||
|
|
||||||
|
**Expected behavior**:
|
||||||
|
- Cluster detects node-03 offline within heartbeat timeout
|
||||||
|
- OVN tunnels from/to node-03 fail (Geneve encap packets lost)
|
||||||
|
- Cluster continues operating with 2/3 quorum
|
||||||
|
- On reconnection: node-03 rejoins, tunnels re-establish, workloads
|
||||||
|
can be restored
|
||||||
|
|
||||||
|
**Recovery**:
|
||||||
|
- If node-03 does not rejoin: check OVN service status, restart if needed
|
||||||
|
- If cluster state is inconsistent: restore from Proxmox snapshot
|
||||||
|
|
||||||
|
**Status**: Defined, not yet executed.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Exercise 4: Full Node Failure
|
||||||
|
|
||||||
|
**Goal**: Verify that the cluster survives a complete node loss and
|
||||||
|
maintains operations with 2/3 quorum. Measure cluster rejoin time
|
||||||
|
after node recovery.
|
||||||
|
|
||||||
|
**Prerequisites**:
|
||||||
|
- Proxmox snapshot of node-03 (VMID 902)
|
||||||
|
- Evacuate ALL workloads from node-03
|
||||||
|
- Verify cluster health: `incusos-health --all`
|
||||||
|
|
||||||
|
**Steps**:
|
||||||
|
1. Evacuate workloads: `incus cluster evacuate oc-node-03`
|
||||||
|
2. Take Proxmox snapshot of node-03
|
||||||
|
3. Hard-stop node-03 via Proxmox:
|
||||||
|
`incusos/helpers/proxmox-api POST /nodes/pve/qemu/902/status/stop`
|
||||||
|
4. Verify cluster continues operating:
|
||||||
|
- `incus cluster list` should show 2 online, 1 offline
|
||||||
|
- Existing workloads on node-01/02 remain accessible
|
||||||
|
- OVN gateway should failover if node-03 was a gateway
|
||||||
|
5. Wait 5 minutes, observing Grafana metrics
|
||||||
|
6. Restart node-03:
|
||||||
|
`incusos/helpers/proxmox-api POST /nodes/pve/qemu/902/status/start`
|
||||||
|
7. Monitor boot: `incusos/helpers/proxmox-screenshot 902`
|
||||||
|
8. Measure time from start to cluster rejoin (node shows online)
|
||||||
|
9. Restore workloads: `incus cluster restore oc-node-03`
|
||||||
|
|
||||||
|
**Expected behavior**:
|
||||||
|
- 2/3 quorum maintained -- all cluster operations continue
|
||||||
|
- OVN gateway failover occurs if node-03 was elected gateway
|
||||||
|
- After restart: node-03 boots, TPM unlocks, Incus starts, rejoins cluster
|
||||||
|
- Typical rejoin time: 1-3 minutes after boot
|
||||||
|
|
||||||
|
**Safety**:
|
||||||
|
- This is NOT a first-boot scenario -- hard-stop is safe for already-
|
||||||
|
provisioned nodes (TPM state is already sealed)
|
||||||
|
- One node at a time ONLY
|
||||||
|
- If node-03 fails to rejoin after restart, restore snapshot
|
||||||
|
|
||||||
|
**Status**: Defined, not yet executed.
|
||||||
|
|
||||||
|
## Safety Rules
|
||||||
|
|
||||||
|
These rules are non-negotiable for all break-fix exercises:
|
||||||
|
|
||||||
|
1. **Never hard-stop during first boot.** First boot seals TPM measurements
|
||||||
|
and writes encryption keys. Interrupting this corrupts TPM state
|
||||||
|
permanently, requiring full reinstallation.
|
||||||
|
|
||||||
|
2. **One node at a time.** A 3-node cluster requires 2/3 quorum. Never
|
||||||
|
take down more than one node simultaneously, or the cluster loses
|
||||||
|
quorum and all operations halt.
|
||||||
|
|
||||||
|
3. **Proxmox snapshots before every destructive test.** Snapshot the
|
||||||
|
target node's VM (VMID 900-939) before any exercise that involves
|
||||||
|
stopping, disconnecting, or modifying the node.
|
||||||
|
|
||||||
|
4. **Verify cluster health before and after.** Run `incusos-health --all`
|
||||||
|
(or equivalent API checks) before starting an exercise and after
|
||||||
|
completing it. Do not proceed if the cluster is already degraded.
|
||||||
|
|
||||||
|
5. **Monitor via Grafana during all exercises.** Visual monitoring catches
|
||||||
|
issues that API polling might miss (e.g., OVN tunnel flapping,
|
||||||
|
storage I/O spikes).
|
||||||
|
|
||||||
|
6. **Target node-03 for destructive tests.** node-03 (VMID 902) is the
|
||||||
|
preferred target because it is not the cluster leader and typically
|
||||||
|
has the fewest workloads. Evacuate before testing.
|
||||||
|
|
||||||
|
7. **Keep recovery keys accessible.** The encryption recovery key and
|
||||||
|
ZFS pool recovery key should be retrieved and stored securely before
|
||||||
|
any exercise that might corrupt TPM state.
|
||||||
|
|
||||||
|
## Helper Script: incusos-health
|
||||||
|
|
||||||
|
The `incusos/helpers/incusos-health` script queries the IncusOS API on
|
||||||
|
cluster nodes to report system state. It is the primary tool for verifying
|
||||||
|
cluster health before and after break-fix exercises.
|
||||||
|
|
||||||
|
### Usage
|
||||||
|
|
||||||
|
```bash
|
||||||
|
incusos/helpers/incusos-health [ACTION] [OPTIONS]
|
||||||
|
```
|
||||||
|
|
||||||
|
### Actions
|
||||||
|
|
||||||
|
| Action | Description |
|
||||||
|
|--------|-------------|
|
||||||
|
| `--status` | Basic system info: version, hostname, TPM status, Secure Boot |
|
||||||
|
| `--partitions` | Disk and partition layout, A/B partition state |
|
||||||
|
| `--tpm` | TPM details, Secure Boot certificates, encryption keys |
|
||||||
|
| `--services` | Enabled/disabled services (OVN, iSCSI, LVM, etc.) |
|
||||||
|
| `--network` | Network interface configuration, DNS, gateway |
|
||||||
|
| `--update` | Update channel, check frequency, pending update status |
|
||||||
|
| `--all` | Run all of the above in sequence |
|
||||||
|
|
||||||
|
### Example output workflow
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Before exercise: verify all nodes healthy
|
||||||
|
incusos/helpers/incusos-health --all
|
||||||
|
|
||||||
|
# After exercise: verify recovery
|
||||||
|
incusos/helpers/incusos-health --status # Quick version check
|
||||||
|
incusos/helpers/incusos-health --all # Full health report
|
||||||
|
```
|
||||||
|
|
||||||
|
## API Reference
|
||||||
|
|
||||||
|
All IncusOS system information is available via the REST API on each node.
|
||||||
|
|
||||||
|
| Endpoint | Returns |
|
||||||
|
|----------|---------|
|
||||||
|
| `/os/1.0` | Version, hostname, basic system info |
|
||||||
|
| `/os/1.0/system/security` | TPM status, Secure Boot, encryption keys |
|
||||||
|
| `/os/1.0/system/storage` | Disks, partitions, ZFS pools |
|
||||||
|
| `/os/1.0/system/resources` | CPU, memory, hardware info |
|
||||||
|
| `/os/1.0/system/network` | Interfaces, DNS, routes |
|
||||||
|
| `/os/1.0/system/update` | Update channel, version, pending updates |
|
||||||
|
| `/os/1.0/services/ovn` | OVN configuration and status |
|
||||||
|
| `/os/1.0/services/iscsi` | iSCSI configuration |
|
||||||
|
| `/os/1.0/services/lvm` | LVM configuration |
|
||||||
|
| `/os/1.0/services/multipath` | Multipath configuration |
|
||||||
|
| `/os/1.0/services/nvme` | NVMe-oF configuration |
|
||||||
|
| `/os/1.0/services/tailscale` | Tailscale VPN configuration |
|
||||||
|
| `/os/1.0/services/usbip` | USB/IP configuration |
|
||||||
|
|
||||||
|
The API listens on the management interface, HTTPS, port 8443 (same as
|
||||||
|
Incus). Authentication uses the Incus client certificate.
|
||||||
Loading…
Reference in New Issue