maarten
/
incus-contrib
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Add macOS support, multi-lab lifecycle, extended testing, and migration docs 

Fix OC reboot diagnosis across all docs (dqlite corruption, not TPM PCR).
Add cross-platform support: genisoimage/mkisofs detection, portable dd/grep/
date usage. Add --lab-up/--lab-down/--resources/--labs to incusos-proxmox for
multi-lab coexistence. Add 7 new lab-test phases (storage, network, projects,
backup, limits, security, resilience). Fix VMID collision in lab-advanced
(910→930). Create migration guide and UTM design doc.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Maarten 2026-02-20 23:41:06 +01:00
parent 8ae8700346
commit f812e04746
13 changed files with 2118 additions and 78 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

59
CLAUDE.md
View File

@ -16,15 +16,17 @@ incu-contrib/
├── incusos/ # IncusOS installation tooling ├── incusos/ # IncusOS installation tooling
│ ├── README.md # Detailed usage docs │ ├── README.md # Detailed usage docs
│ ├── incusos-iso # ISO/IMG builder (wraps flasher-tool) │ ├── incusos-iso # ISO/IMG builder (wraps flasher-tool)
│ ├── incusos-seed # Seed archive generator │ ├── incusos-seed # Seed archive generator (cross-platform: Linux + macOS)
│ ├── incusos-proxmox # Declarative Proxmox VM deployment │ ├── incusos-proxmox # Declarative Proxmox VM deployment + lab lifecycle
│ ├── lab-test # Guided lab validation (workloads, clustering, migration) │ ├── lab-test # Guided lab validation (12 test phases)
│ ├── proxmox.yaml # Proxmox connection config (gitignored, contains credentials) │ ├── proxmox.yaml # Proxmox connection config (gitignored, contains credentials)
│ ├── TESTING.md # Testing guide for incusos-proxmox and lab-test │ ├── TESTING.md # Testing guide for incusos-proxmox and lab-test
│ └── examples/ # Example seed + Proxmox YAML files │ └── examples/ # Example seed + Proxmox YAML files
└── notes/ # Research notes and reference material └── notes/ # Research notes and reference material
├── clustering-guide.md # Detailed Incus clustering walkthrough ├── clustering-guide.md # Detailed Incus clustering walkthrough
└── operations-center-guide.md # Operations Center provisioning & management ├── operations-center-guide.md # Operations Center provisioning & management
├── migration-guide.md # Migration paths into Incus from other hypervisors
└── utm-support.md # UTM support design document (future)
``` ```
## Key technical context ## Key technical context
@ -146,6 +148,28 @@ incu-contrib/
acting -- already-running VMs are skipped, stopped-but-installed VMs are acting -- already-running VMs are skipped, stopped-but-installed VMs are
started from disk, only VMs with install media proceed through installation. started from disk, only VMs with install media proceed through installation.
### Multi-lab coexistence
- **Lab lifecycle**: `--lab-down config.yaml` stops all VMs (Proxmox stop,
VMs stay on disk). `--lab-up config.yaml` starts stopped VMs from disk
(refuses to start VMs with install media still attached). These are
distinct from `--cleanup` (which destroys VMs permanently).
- **Resource awareness**: `--resources` shows Proxmox host RAM, CPU, storage
usage, and per-pool allocation. Requires API method.
- **Lab inventory**: `--labs` scans the pool for managed VMs (by
`[incusos-lab:managed]` marker), groups by config file, and shows
per-lab status, VM count, RAM, and disk.
- **VMID range convention** (to avoid collisions between coexisting labs):
| Range | Lab |
|-------|-----|
| 400-499 | OC-managed nodes |
| 800-809 | Single-node labs |
| 900-909 | Basic cluster |
| 910-919 | OC combined (server + nodes) |
| 920-929 | OC server standalone |
| 930-939 | Advanced / heterogeneous |
### Incus clustering via remotes ### Incus clustering via remotes
- **Cluster formation** is done entirely through the `incus` CLI using remotes. - **Cluster formation** is done entirely through the `incus` CLI using remotes.
@ -399,9 +423,10 @@ incu-contrib/
#### Tested limitations #### Tested limitations
- **Inventory is NOT real-time** -- requires explicit `cluster resync` - **Inventory is NOT real-time** -- requires explicit `cluster resync`
- **OC reboot BREAKS IncusOS on Proxmox** -- guest reboot corrupts TPM - **OC reboot BREAKS IncusOS on Proxmox** -- guest reboot likely corrupts
encryption. Node becomes unreachable. Only fix: destroy and redeploy. the dqlite/Incus database through unclean shutdown. OS boots (pingable)
Use Proxmox stop/start instead. but Incus daemon never starts. Open research item. Only fix: destroy
and redeploy. Use Proxmox stop/start instead.
- **No cluster member state tracking** -- OC always shows `ready` even for - **No cluster member state tracking** -- OC always shows `ready` even for
EVACUATED/OFFLINE nodes EVACUATED/OFFLINE nodes
- **Stale entries** from out-of-band cluster changes persist after resync - **Stale entries** from out-of-band cluster changes persist after resync
@ -440,6 +465,26 @@ incu-contrib/
- **No hardcoded package managers**: say "install the Incus client" with a link, - **No hardcoded package managers**: say "install the Incus client" with a link,
not "sudo apt install incus". not "sudo apt install incus".
### Migration into Incus
- **`incus-migrate`**: official tool for importing disk images, running
instances, or physical machines into Incus.
- **Disk format conversion**: use `qemu-img convert` between vmdk, qcow2,
raw, vdi, vhd formats. Incus accepts raw and qcow2.
- **Import workflow**: convert disk → `incus storage volume import` →
`incus init --empty --vm` → attach disk → start.
- **Container migration**: `docker export` → `incus import` for filesystem-
level container migration. Docker volumes must be copied separately.
- See `notes/migration-guide.md` for full procedures per source hypervisor.
### UTM support (future)
- Design document at `notes/utm-support.md`.
- UTM provides `utmctl` CLI for start/stop/status but **not** for VM creation
(requires AppleScript or `.utm` bundle generation).
- No `blockstat` equivalent -- install detection must use timeout + port polling.
- Seed generation already works cross-platform (Phase 3 macOS compatibility).
## Git workflow ## Git workflow
- Main branch: `main` - Main branch: `main`

8
README.md
View File

@ -34,6 +34,14 @@ Research notes and reference material:
install instructions, and certificate management notes. install instructions, and certificate management notes.
- [`iso-download-methods.md`](notes/iso-download-methods.md) -- - [`iso-download-methods.md`](notes/iso-download-methods.md) --
Original research on ISO download/customization approaches. Original research on ISO download/customization approaches.
- [`clustering-guide.md`](notes/clustering-guide.md) --
Detailed Incus clustering walkthrough (manual cluster formation via remotes).
- [`operations-center-guide.md`](notes/operations-center-guide.md) --
Operations Center provisioning, cluster management, and tested limitations.
- [`migration-guide.md`](notes/migration-guide.md) --
Migration paths into Incus from Proxmox, UTM, VMware Fusion, and Docker.
- [`utm-support.md`](notes/utm-support.md) --
Design document for future UTM support on macOS.
## Quick Start ## Quick Start

59
incusos/README.md
View File

@ -493,6 +493,29 @@ With `--yes`, defaults to option 1 (safe — never auto-destroys).
already-running VMs are skipped, stopped-but-installed VMs are started from disk, already-running VMs are skipped, stopped-but-installed VMs are started from disk,
and only VMs with install media still attached proceed through installation. and only VMs with install media still attached proceed through installation.
### Multi-Lab Management
Multiple lab environments can coexist on the same Proxmox host. Each lab uses
a non-overlapping VMID range (see `start_vmid` in example configs). Only one
lab needs to be running at a time — the others can be stopped to save RAM.
```bash
# Stop a lab (VMs stay on disk, ready to restart)
./incusos-proxmox --lab-down examples/lab-cluster.yaml
# Start a different lab
./incusos-proxmox --lab-up examples/lab-advanced.yaml
# See all deployed labs and their status
./incusos-proxmox --labs
# Check Proxmox host resource usage (RAM, storage, per-pool summary)
./incusos-proxmox --resources
# Permanently destroy a lab
./incusos-proxmox --cleanup examples/lab-cluster.yaml
```
### Proxmox VM Settings ### Proxmox VM Settings
Every VM is created with these IncusOS-specific settings: Every VM is created with these IncusOS-specific settings:
@ -511,7 +534,7 @@ Every VM is created with these IncusOS-specific settings:
### Prerequisites ### Prerequisites
- **python3** + **PyYAML** (`python3-yaml`) -- YAML config parsing - **python3** + **PyYAML** (`python3-yaml`) -- YAML config parsing
- **genisoimage** -- for creating ISO 9660 seed images - **genisoimage** (Linux) or **mkisofs** from cdrtools (macOS) -- for ISO 9660 seed images
- **ssh** + **scp** (for SSH method) or **curl** (for API method) - **ssh** + **scp** (for SSH method) or **curl** (for API method)
- **incusos-seed** -- sibling script (auto-detected in same directory) - **incusos-seed** -- sibling script (auto-detected in same directory)
- **Bash 4+** -- associative arrays - **Bash 4+** -- associative arrays
@ -639,7 +662,41 @@ See the [`examples/`](examples/) directory for sample seed configurations:
- [`proxmox-lab.yaml`](examples/proxmox-lab.yaml) -- Proxmox: 4-VM lab (self-contained, includes proxmox: section) - [`proxmox-lab.yaml`](examples/proxmox-lab.yaml) -- Proxmox: 4-VM lab (self-contained, includes proxmox: section)
- [`proxmox-minimal.yaml`](examples/proxmox-minimal.yaml) -- Proxmox: single standalone VM (self-contained) - [`proxmox-minimal.yaml`](examples/proxmox-minimal.yaml) -- Proxmox: single standalone VM (self-contained)
- [`lab-cluster.yaml`](examples/lab-cluster.yaml) -- Proxmox: 3-node cluster (uses proxmox.yaml) - [`lab-cluster.yaml`](examples/lab-cluster.yaml) -- Proxmox: 3-node cluster (uses proxmox.yaml)
- [`lab-single.yaml`](examples/lab-single.yaml) -- Proxmox: single standalone node (uses proxmox.yaml)
- [`lab-advanced.yaml`](examples/lab-advanced.yaml) -- Proxmox: 3-node heterogeneous migration cluster
- [`lab-oc.yaml`](examples/lab-oc.yaml) -- Proxmox: Operations Center + 3 nodes (uses proxmox.yaml) - [`lab-oc.yaml`](examples/lab-oc.yaml) -- Proxmox: Operations Center + 3 nodes (uses proxmox.yaml)
- [`lab-oc-deploy.yaml`](examples/lab-oc-deploy.yaml) -- Proxmox: OC server only (for hybrid OC workflow)
- [`lab-oc-nodes.yaml`](examples/lab-oc-nodes.yaml) -- Proxmox: 3 OC-managed nodes (requires `--iso`)
## Platform Support
| Script | Linux | macOS | Notes |
|--------|:-----:|:-----:|-------|
| `incusos-iso` | yes | yes | Requires flasher-tool (Go binary, compiles on both) and genisoimage/mkisofs |
| `incusos-seed` | yes | yes | Requires genisoimage (Linux) or mkisofs from cdrtools (macOS) for ISO format |
| `incusos-proxmox` | yes | yes | Uses incusos-seed for seed generation; SSH/API transport is cross-platform |
| `lab-test` | yes | yes | Only needs the `incus` CLI client |
**macOS prerequisites** (when running tools against a remote Proxmox host):
```bash
brew install cdrtools # provides mkisofs (replaces genisoimage)
brew install mtools # provides mcopy (for FAT seed images)
brew install incus # Incus CLI client
```
Scripts auto-detect the platform and use the appropriate tools (e.g. `mkisofs`
on macOS vs `genisoimage` on Linux).
## Future: UTM Support
A future `incusos-utm` script could deploy IncusOS VMs on macOS using UTM
as the hypervisor. The seed generation (`incusos-seed`) already works on macOS.
The main challenge is VM creation (UTM's `utmctl` CLI doesn't support it --
AppleScript or `.utm` bundle generation would be needed).
See [`notes/utm-support.md`](../notes/utm-support.md) for the full design
document.
## Security Notes ## Security Notes

29
incusos/TESTING.md
View File

@ -398,8 +398,29 @@ incus remote add incus-lab-03 <IP3> --accept-certificate
# Migration and evacuation tests # Migration and evacuation tests
./incusos/lab-test --phase migrate --skip-deploy examples/lab-cluster.yaml ./incusos/lab-test --phase migrate --skip-deploy examples/lab-cluster.yaml
# Extended phases (require a working cluster):
./incusos/lab-test --phase storage --skip-deploy examples/lab-cluster.yaml
./incusos/lab-test --phase network --skip-deploy examples/lab-cluster.yaml
./incusos/lab-test --phase projects --skip-deploy examples/lab-cluster.yaml
./incusos/lab-test --phase backup --skip-deploy examples/lab-cluster.yaml
./incusos/lab-test --phase limits --skip-deploy examples/lab-cluster.yaml
./incusos/lab-test --phase security --skip-deploy examples/lab-cluster.yaml
./incusos/lab-test --phase resilience --skip-deploy examples/lab-cluster.yaml
``` ```
### Extended Test Phases
| Phase | Tests | Requires |
|-------|-------|----------|
| `storage` | Pool listing, volume create/delete, volume snapshots, pool usage | Single node |
| `network` | Network listing, bridge create/delete, ACLs, forwards | Single node |
| `projects` | Project create with limits, custom profiles, profile inheritance | Single node |
| `backup` | Instance snapshots (create/restore/delete), export/import round-trip | Single node |
| `limits` | CPU limits, memory limits, disk I/O limits | Single node |
| `security` | Unprivileged vs privileged containers, nesting, user namespace isolation | Single node |
| `resilience` | Cluster health, database leader, member state, warnings | 3-node cluster |
### Manual Cluster Formation (Alternative) ### Manual Cluster Formation (Alternative)
If you prefer to form the cluster manually instead of using `lab-test`, If you prefer to form the cluster manually instead of using `lab-test`,
@ -812,8 +833,10 @@ incus cluster restore oc-node-01:oc-node-01 --force
``` ```
**WARNING**: do NOT use `operations-center provisioning server system reboot`. **WARNING**: do NOT use `operations-center provisioning server system reboot`.
It sends a guest-level reboot that corrupts TPM disk encryption on Proxmox VMs. It sends a guest-level reboot that likely corrupts the dqlite/Incus database
The node becomes unreachable and must be destroyed/redeployed. through unclean shutdown on Proxmox VMs. The node becomes pingable (OS boots)
but Incus daemon never starts (port 8443 closed). This is an open research
item. The node must be destroyed/redeployed.
### Cleanup ### Cleanup
@ -866,7 +889,7 @@ operations-center remote remove oc-lab 2>/dev/null
| Bash mangles Proxmox API token with `!` | History expansion | Use single quotes around token value or store in variable | | Bash mangles Proxmox API token with `!` | History expansion | Use single quotes around token value or store in variable |
| OC token seed fields all `{}` | Flat YAML format used | Use structured format: `install: { force_reboot: true }` not `force_reboot: true` at root | | OC token seed fields all `{}` | Flat YAML format used | Use structured format: `install: { force_reboot: true }` not `force_reboot: true` at root |
| OC cluster Terraform errors ("already in trust store") | `apply_defaults: true` pre-created resources | Non-fatal -- cluster forms despite errors. Skip `--application-seed-config` or skip `apply_defaults` | | OC cluster Terraform errors ("already in trust store") | `apply_defaults: true` pre-created resources | Non-fatal -- cluster forms despite errors. Skip `--application-seed-config` or skip `apply_defaults` |
| OC reboot kills IncusOS node (port 8443 closed forever) | Guest reboot corrupts TPM PCR measurements | Destroy and redeploy VM. Use Proxmox stop/start instead of OC reboot | | OC reboot kills IncusOS node (port 8443 closed forever) | Likely dqlite/Incus database corruption from unclean shutdown (open research item) | Destroy and redeploy VM. Use Proxmox stop/start instead of OC reboot |
| OC inventory empty after creating workloads | Inventory not real-time | Run `operations-center provisioning cluster resync <name>` | | OC inventory empty after creating workloads | Inventory not real-time | Run `operations-center provisioning cluster resync <name>` |
| OC shows stale server entry after node replacement | Out-of-band cluster change | OC can't detect `incus cluster remove`; stale entries persist | | OC shows stale server entry after node replacement | Out-of-band cluster change | OC can't detect `incus cluster remove`; stale entries persist |
| OC `server remove` fails ("part of cluster") | Server is in OC cluster | Must remove cluster first or handle via OC cluster management | | OC `server remove` fails ("part of cluster") | Server is in OC cluster | Must remove cluster first or handle via OC cluster management |

6
incusos/examples/lab-advanced.yaml
View File

@ -20,13 +20,13 @@
# Connection settings from proxmox.yaml (see examples/proxmox.yaml.example). # Connection settings from proxmox.yaml (see examples/proxmox.yaml.example).
# Add a proxmox: section here to override per-deployment. # Add a proxmox: section here to override per-deployment.
# #
# Note: start_vmid 910 may collide with lab-oc.yaml. Clean up old deployments # VMID range 930-939 reserved for advanced/heterogeneous labs.
# first or adjust the range. # See VMID range table in CLAUDE.md to avoid collisions with other labs.
defaults: defaults:
memory: 8192 memory: 8192
disk: 50 disk: 50
start_vmid: 910 start_vmid: 930
vms: vms:
- name: incus-adv-01 - name: incus-adv-01

25
incusos/examples/lab-single.yaml Normal file
View File

@ -0,0 +1,25 @@
# lab-single.yaml - Single IncusOS node for basic testing
#
# Deploys 1 standalone Incus node with defaults. Useful for:
# - Testing single-node workloads (containers, VMs)
# - Verifying seed configs and install flow
# - Quick experiments without a full cluster
#
# Usage:
# incusos-proxmox --dry-run examples/lab-single.yaml # Preview
# incusos-proxmox --yes examples/lab-single.yaml # Deploy
# incusos-proxmox --status examples/lab-single.yaml # Check status
# incusos-proxmox --cleanup examples/lab-single.yaml # Tear down
#
# Connection settings from proxmox.yaml (see examples/proxmox.yaml.example).
defaults:
cores: 4
memory: 4096
disk: 50
start_vmid: 800
vms:
- name: incus-single
app: incus
apply_defaults: true

3
incusos/incusos-iso
View File

@ -502,7 +502,8 @@ detect_client_cert() {
fi fi
# CLI is installed but cert retrieval failed -- check why # CLI is installed but cert retrieval failed -- check why
local incus_version local incus_version
incus_version=$(incus version 2>/dev/null | grep -oP 'Client version: \K[0-9.]+' || echo "unknown") incus_version=$(incus version 2>/dev/null | sed -n 's/.*Client version: \([0-9.]*\).*/\1/p')
[[ -z "$incus_version" ]] && incus_version="unknown"
detail "Incus CLI v${incus_version} found but no certificate available" detail "Incus CLI v${incus_version} found but no certificate available"
detail "Generate one with: incus remote list (triggers auto-generation)" detail "Generate one with: incus remote list (triggers auto-generation)"
fi fi

460
incusos/incusos-proxmox
View File

@ -93,6 +93,10 @@ ${BOLD}OPTIONS${RESET}
${BOLD}--cleanup-all${RESET} Destroy ALL managed VMs in the pool ${BOLD}--cleanup-all${RESET} Destroy ALL managed VMs in the pool
${BOLD}--proxmox${RESET} FILE Proxmox connection config (default: auto-discover) ${BOLD}--proxmox${RESET} FILE Proxmox connection config (default: auto-discover)
${BOLD}--force-cleanup${RESET} Destroy VMs even without management marker ${BOLD}--force-cleanup${RESET} Destroy VMs even without management marker
${BOLD}--lab-up${RESET} Start all stopped VMs from config (boot from disk)
${BOLD}--lab-down${RESET} Stop all running VMs from config (Proxmox stop)
${BOLD}--resources${RESET} Show Proxmox host resource usage (RAM, storage)
${BOLD}--labs${RESET} List all deployed labs (scan pool for managed VMs)
${BOLD}--dry-run${RESET} Preview actions without executing ${BOLD}--dry-run${RESET} Preview actions without executing
${BOLD}-v, --verbose${RESET} Show detailed output (tool paths, API calls) ${BOLD}-v, --verbose${RESET} Show detailed output (tool paths, API calls)
${BOLD}-q, --quiet${RESET} Suppress informational output ${BOLD}-q, --quiet${RESET} Suppress informational output
@ -244,6 +248,10 @@ CLEANUP=false
CLEANUP_ALL=false CLEANUP_ALL=false
DEEP_CLEANUP=false DEEP_CLEANUP=false
FORCE_CLEANUP=false FORCE_CLEANUP=false
LAB_UP=false
LAB_DOWN=false
RESOURCES=false
LIST_LABS=false
DRY_RUN=false DRY_RUN=false
QUIET=false QUIET=false
VERBOSE=false VERBOSE=false
@ -310,6 +318,22 @@ parse_args() {
FORCE_CLEANUP=true FORCE_CLEANUP=true
shift shift
;; ;;
--lab-up)
LAB_UP=true
shift
;;
--lab-down)
LAB_DOWN=true
shift
;;
--resources)
RESOURCES=true
shift
;;
--labs)
LIST_LABS=true
shift
;;
--proxmox) --proxmox)
PROXMOX_CONFIG="${2:?'--proxmox requires a file path'}" PROXMOX_CONFIG="${2:?'--proxmox requires a file path'}"
shift 2 shift 2
@ -368,10 +392,16 @@ validate_args() {
return 0 return 0
fi fi
# --cleanup-all can run without a config file (uses proxmox.yaml) # --cleanup-all, --labs, --resources can run without a config file (uses proxmox.yaml)
if [[ "$CLEANUP_ALL" == true ]] && [[ -z "$CONFIG_FILE" ]]; then if [[ "$CLEANUP_ALL" == true ]] && [[ -z "$CONFIG_FILE" ]]; then
return 0 return 0
fi fi
if [[ "$LIST_LABS" == true ]] && [[ -z "$CONFIG_FILE" ]]; then
return 0
fi
if [[ "$RESOURCES" == true ]] && [[ -z "$CONFIG_FILE" ]]; then
return 0
fi
if [[ -z "$CONFIG_FILE" ]]; then if [[ -z "$CONFIG_FILE" ]]; then
error "No configuration file specified" error "No configuration file specified"
@ -522,13 +552,20 @@ run_doctor() {
local script_dir local script_dir
script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# -- Platform section --
echo -e " ${BOLD}Platform:${RESET}"
echo -e " OS: $(uname -s) $(uname -m)"
echo -e " Bash: ${BASH_VERSION}"
echo ""
# -- Tools section -- # -- Tools section --
echo -e " ${BOLD}Tools:${RESET}" echo -e " ${BOLD}Tools:${RESET}"
# incus client # incus client
if command -v incus &>/dev/null; then if command -v incus &>/dev/null; then
local incus_ver local incus_ver
incus_ver=$(incus version 2>/dev/null | grep -oP 'Client version: \K[0-9.]+' || echo "unknown") incus_ver=$(incus version 2>/dev/null | sed -n 's/.*Client version: \([0-9.]*\).*/\1/p')
[[ -z "$incus_ver" ]] && incus_ver="unknown"
echo -e " incus client: ${GREEN}${incus_ver} ✓${RESET}" echo -e " incus client: ${GREEN}${incus_ver} ✓${RESET}"
else else
echo -e " incus client: ${YELLOW}not installed${RESET}" echo -e " incus client: ${YELLOW}not installed${RESET}"
@ -573,11 +610,16 @@ run_doctor() {
echo -e " curl: ${RED}not installed (required)${RESET}" echo -e " curl: ${RED}not installed (required)${RESET}"
fi fi
# genisoimage # ISO creation tool (genisoimage on Linux, mkisofs on macOS)
if command -v genisoimage &>/dev/null; then if command -v genisoimage &>/dev/null; then
echo -e " genisoimage: ${GREEN}installed ✓${RESET}" echo -e " genisoimage: ${GREEN}installed ✓${RESET}"
elif command -v mkisofs &>/dev/null; then
echo -e " mkisofs: ${GREEN}installed ✓${RESET} (cdrtools)"
else else
echo -e " genisoimage: ${RED}not installed (required for seed ISOs)${RESET}" case "$(uname -s)" in
Darwin) echo -e " mkisofs: ${RED}not installed (brew install cdrtools)${RESET}" ;;
*) echo -e " genisoimage: ${RED}not installed (required for seed ISOs)${RESET}" ;;
esac
fi fi
# jq # jq
@ -2317,10 +2359,13 @@ for s in data.get('data', []):
fi fi
fi fi
# 5. genisoimage for SEED_DATA ISO 9660 generation # 5. ISO creation tool for SEED_DATA ISO 9660 generation
if ! command -v genisoimage &>/dev/null; then if ! command -v genisoimage &>/dev/null && ! command -v mkisofs &>/dev/null; then
error "genisoimage is required for SEED_DATA generation but not found" error "An ISO creation tool is required for SEED_DATA generation but not found"
error "Install with: sudo apt install genisoimage" case "$(uname -s)" in
Darwin) error "Install with: brew install cdrtools" ;;
*) error "Install with: sudo apt install genisoimage" ;;
esac
errors=$((errors + 1)) errors=$((errors + 1))
fi fi
@ -3196,6 +3241,359 @@ phase_install() {
success "Installation phase complete" success "Installation phase complete"
} }
# ---------------------------------------------------------------------------
# Lab lifecycle (--lab-up, --lab-down)
# ---------------------------------------------------------------------------
phase_lab_down() {
step "Stopping lab VMs"
echo ""
local count=${#VM_NAMES[@]}
local stopped=0
local skipped=0
local i=0
while [[ $i -lt $count ]]; do
local name="${VM_NAMES[$i]}"
local vmid="${VM_VMIDS[$i]}"
if [[ "$DRY_RUN" == true ]]; then
info "Would stop ${name} (VMID ${vmid})"
i=$((i + 1))
continue
fi
if ! pve_vm_exists "$vmid"; then
warn "${name} (VMID ${vmid}): not found -- skipping"
skipped=$((skipped + 1))
i=$((i + 1))
continue
fi
local status
status=$(pve_vm_status "$vmid") || status="unknown"
if [[ "$status" == "running" ]]; then
info "Stopping ${name} (VMID ${vmid})..."
if pve_stop_vm "$vmid"; then
success "${name} stopped"
stopped=$((stopped + 1))
else
error "Failed to stop ${name}"
fi
else
detail "${name}: already ${status}"
skipped=$((skipped + 1))
fi
i=$((i + 1))
done
echo ""
if [[ "$DRY_RUN" == true ]]; then
info "Dry run -- no VMs were stopped"
else
success "Lab down: ${stopped} stopped, ${skipped} skipped"
fi
}
phase_lab_up() {
step "Starting lab VMs"
echo ""
local count=${#VM_NAMES[@]}
local started=0
local skipped=0
local i=0
while [[ $i -lt $count ]]; do
local name="${VM_NAMES[$i]}"
local vmid="${VM_VMIDS[$i]}"
if [[ "$DRY_RUN" == true ]]; then
info "Would start ${name} (VMID ${vmid})"
i=$((i + 1))
continue
fi
if ! pve_vm_exists "$vmid"; then
warn "${name} (VMID ${vmid}): not found -- skipping"
skipped=$((skipped + 1))
i=$((i + 1))
continue
fi
local status
status=$(pve_vm_status "$vmid") || status="unknown"
if [[ "$status" == "stopped" ]]; then
# Safety check: refuse to start VMs with install media still attached
if pve_vm_has_install_media "$vmid"; then
warn "${name}: install media still attached -- skipping (run full deploy instead)"
skipped=$((skipped + 1))
i=$((i + 1))
continue
fi
info "Starting ${name} (VMID ${vmid})..."
if pve_start_vm "$vmid"; then
success "${name} started"
started=$((started + 1))
else
error "Failed to start ${name}"
fi
else
detail "${name}: already ${status}"
skipped=$((skipped + 1))
fi
i=$((i + 1))
done
if [[ "$DRY_RUN" == true ]]; then
echo ""
info "Dry run -- no VMs were started"
return
fi
echo ""
success "Lab up: ${started} started, ${skipped} skipped"
if [[ $started -gt 0 ]]; then
info "Waiting 30s for services to start..."
sleep 30
echo ""
phase_status
fi
}
# ---------------------------------------------------------------------------
# Resource and lab inventory (--resources, --labs)
# ---------------------------------------------------------------------------
do_resources() {
echo -e "${BOLD}${SCRIPT_NAME} resources${RESET}"
echo ""
echo -e " ${BOLD}Proxmox Host:${RESET} ${PVE_HOST} (node: ${PVE_NODE})"
echo ""
# Host resources (API only -- SSH would need parsing pvesh output)
if [[ "$PVE_METHOD" != "api" ]]; then
warn "Resource details require API method (current: ${PVE_METHOD})"
warn "Use --method api or set method: api in proxmox.yaml"
return
fi
# Host memory and CPU
local node_status
node_status=$(pve_api GET "/nodes/${PVE_NODE}/status" 2>/dev/null) || node_status=""
if [[ -n "$node_status" ]]; then
python3 -c "
import json, sys
data = json.loads('''${node_status}''').get('data', {})
mem = data.get('memory', {})
cpu_count = data.get('cpuinfo', {}).get('cpus', '?')
cpu_model = data.get('cpuinfo', {}).get('model', 'unknown')
mem_used = mem.get('used', 0) / (1024**3)
mem_total = mem.get('total', 0) / (1024**3)
mem_free = mem_total - mem_used
mem_pct = (mem_used / mem_total * 100) if mem_total > 0 else 0
print(f' RAM: {mem_used:.1f} GiB / {mem_total:.1f} GiB used ({mem_pct:.0f}%), {mem_free:.1f} GiB free')
print(f' CPUs: {cpu_count} cores ({cpu_model})')
uptime_s = data.get('uptime', 0)
days = uptime_s // 86400
hours = (uptime_s % 86400) // 3600
print(f' Uptime: {days}d {hours}h')
" 2>/dev/null || warn "Failed to parse node status"
fi
# Storage
echo ""
echo -e " ${BOLD}Storage:${RESET}"
local storage_list
storage_list=$(pve_api GET "/nodes/${PVE_NODE}/storage" 2>/dev/null) || storage_list=""
if [[ -n "$storage_list" ]]; then
python3 -c "
import json, sys
data = json.loads('''${storage_list}''').get('data', [])
for s in sorted(data, key=lambda x: x.get('storage', '')):
name = s.get('storage', '?')
total = s.get('total', 0) / (1024**3)
used = s.get('used', 0) / (1024**3)
avail = s.get('avail', 0) / (1024**3)
pct = (used / total * 100) if total > 0 else 0
stype = s.get('type', '?')
content = s.get('content', '?')
if total > 0:
print(f' {name:<16} {stype:<12} {used:.1f}G / {total:.1f}G ({pct:.0f}%) -- {content}')
else:
print(f' {name:<16} {stype:<12} (no capacity info) -- {content}')
" 2>/dev/null || warn "Failed to parse storage list"
fi
# Per-pool VM summary (if pool is configured)
if [[ -n "$PVE_POOL" ]]; then
echo ""
echo -e " ${BOLD}Pool: ${PVE_POOL}${RESET}"
local pool_data
pool_data=$(pve_api GET "/pools/${PVE_POOL}" 2>/dev/null) || pool_data=""
if [[ -n "$pool_data" ]]; then
python3 -c "
import json, sys
data = json.loads('''${pool_data}''').get('data', {})
members = data.get('members', [])
vms = [m for m in members if m.get('type') == 'qemu']
running = sum(1 for v in vms if v.get('status') == 'running')
stopped = sum(1 for v in vms if v.get('status') == 'stopped')
total_mem = sum(v.get('maxmem', 0) for v in vms) / (1024**3)
total_disk = sum(v.get('maxdisk', 0) for v in vms) / (1024**3)
print(f' VMs: {len(vms)} total ({running} running, {stopped} stopped)')
print(f' Allocated: {total_mem:.1f} GiB RAM, {total_disk:.0f} GiB disk')
" 2>/dev/null || warn "Failed to parse pool data"
fi
fi
echo ""
}
do_list_labs() {
echo -e "${BOLD}${SCRIPT_NAME} labs${RESET}"
echo ""
if [[ "$PVE_METHOD" != "api" ]]; then
warn "Lab listing requires API method (current: ${PVE_METHOD})"
warn "Use --method api or set method: api in proxmox.yaml"
return
fi
# Get all managed VMs and group by config file
local all_vms
if [[ -n "$PVE_POOL" ]]; then
all_vms=$(pve_api GET "/pools/${PVE_POOL}" 2>/dev/null) || all_vms=""
else
all_vms=$(pve_api GET "/nodes/${PVE_NODE}/qemu" 2>/dev/null) || all_vms=""
fi
if [[ -z "$all_vms" ]]; then
info "No VMs found"
return
fi
# Collect managed VMs and group by config= tag in description
local vmids=()
if [[ -n "$PVE_POOL" ]]; then
# Pool response has members[] with type=qemu
mapfile -t vmids < <(python3 -c "
import json
data = json.loads('''${all_vms}''').get('data', {})
for m in data.get('members', []):
if m.get('type') == 'qemu':
print(m.get('vmid', ''))
" 2>/dev/null)
else
mapfile -t vmids < <(python3 -c "
import json
data = json.loads('''${all_vms}''').get('data', [])
for v in data:
print(v.get('vmid', ''))
" 2>/dev/null)
fi
# For each VM, get config and check for managed marker
declare -A lab_configs # config_name -> "vmid1:name1:status1 vmid2:name2:status2 ..."
declare -A lab_ram # config_name -> total RAM in MiB
declare -A lab_disk # config_name -> total disk in GiB
local unmanaged=0
for vmid in "${vmids[@]}"; do
[[ -z "$vmid" ]] && continue
local config_output
config_output=$(pve_api GET "/nodes/${PVE_NODE}/qemu/${vmid}/config" 2>/dev/null) || continue
local desc name mem_mib disk_gib status
read -r desc name mem_mib disk_gib < <(python3 -c "
import json
data = json.loads('''${config_output}''').get('data', {})
desc = data.get('description', '')
name = data.get('name', 'unknown')
mem = int(data.get('memory', 0))
# Rough disk from scsi0 size
scsi0 = data.get('scsi0', '')
disk_g = 0
for part in scsi0.split(','):
if part.strip().startswith('size='):
sz = part.strip().split('=')[1]
if sz.endswith('G'):
disk_g = int(sz[:-1])
print(f'{desc}|||{name}|||{mem}|||{disk_g}')
" 2>/dev/null | tr '|||' ' ' || true)
# Check for managed marker
if [[ "$desc" != *"$MANAGED_MARKER"* ]]; then
unmanaged=$((unmanaged + 1))
continue
fi
# Extract config= tag
local config_tag="unknown"
if [[ "$desc" =~ config=([^\ \]]+) ]]; then
config_tag="${BASH_REMATCH[1]}"
fi
# Get VM status
status=$(pve_vm_status "$vmid") || status="unknown"
# Accumulate
lab_configs["$config_tag"]="${lab_configs[$config_tag]:-} ${vmid}:${name}:${status}"
lab_ram["$config_tag"]=$(( ${lab_ram[$config_tag]:-0} + ${mem_mib:-0} ))
lab_disk["$config_tag"]=$(( ${lab_disk[$config_tag]:-0} + ${disk_gib:-0} ))
done
if [[ ${#lab_configs[@]} -eq 0 ]]; then
info "No managed labs found"
[[ $unmanaged -gt 0 ]] && detail "${unmanaged} unmanaged VMs in pool"
return
fi
for config in $(echo "${!lab_configs[@]}" | tr ' ' '\n' | sort); do
local vms="${lab_configs[$config]}"
local ram_mib="${lab_ram[$config]}"
local disk_gib="${lab_disk[$config]}"
local vm_count=0
local running=0
local stopped=0
echo -e " ${BOLD}${config}${RESET}"
for entry in $vms; do
[[ -z "$entry" ]] && continue
local v_vmid v_name v_status
IFS=':' read -r v_vmid v_name v_status <<< "$entry"
vm_count=$((vm_count + 1))
[[ "$v_status" == "running" ]] && running=$((running + 1))
[[ "$v_status" == "stopped" ]] && stopped=$((stopped + 1))
local status_color="${DIM}"
[[ "$v_status" == "running" ]] && status_color="${GREEN}"
[[ "$v_status" == "stopped" ]] && status_color="${YELLOW}"
echo -e " ${v_name} (${v_vmid}): ${status_color}${v_status}${RESET}"
done
local lab_status="${GREEN}all running${RESET}"
if [[ $running -eq 0 ]] && [[ $stopped -gt 0 ]]; then
lab_status="${YELLOW}all stopped${RESET}"
elif [[ $running -gt 0 ]] && [[ $stopped -gt 0 ]]; then
lab_status="${YELLOW}mixed${RESET}"
fi
echo -e " Status: ${lab_status} | RAM: $((ram_mib / 1024))G | Disk: ${disk_gib}G"
echo ""
done
[[ $unmanaged -gt 0 ]] && detail "${unmanaged} unmanaged VMs in pool"
}
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Status check # Status check
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@ -3951,6 +4349,34 @@ main() {
setup_workdir setup_workdir
# Handle --labs and --resources without a config file (only needs proxmox.yaml)
if { [[ "$LIST_LABS" == true ]] || [[ "$RESOURCES" == true ]]; } && [[ -z "$CONFIG_FILE" ]]; then
if ! find_proxmox_config; then
error "No proxmox.yaml found and no config file specified"
error "Create proxmox.yaml or use --proxmox FILE"
exit 1
fi
load_proxmox_config "$PROXMOX_CONFIG_RESOLVED"
[[ -z "$PVE_NODE" ]] && PVE_NODE="pve"
[[ -z "$PVE_STORAGE" ]] && PVE_STORAGE="local-lvm"
[[ -z "$PVE_ISO_STORAGE" ]] && PVE_ISO_STORAGE="local"
[[ -z "$PVE_BRIDGE" ]] && PVE_BRIDGE="vmbr0"
[[ -z "$PVE_SSH_USER" ]] && PVE_SSH_USER="root"
[[ -z "$PVE_METHOD" ]] && PVE_METHOD="ssh"
[[ -n "$METHOD" ]] && PVE_METHOD="$METHOD"
[[ -n "$HOST_OVERRIDE" ]] && PVE_HOST="$HOST_OVERRIDE"
if [[ -z "$PVE_HOST" ]]; then
error "proxmox.yaml must contain 'host:'"
exit 1
fi
[[ "$RESOURCES" == true ]] && do_resources
[[ "$LIST_LABS" == true ]] && do_list_labs
return
fi
# Handle cleanup-all without a config file (only needs proxmox.yaml) # Handle cleanup-all without a config file (only needs proxmox.yaml)
if [[ "$CLEANUP_ALL" == true ]] && [[ -z "$CONFIG_FILE" ]]; then if [[ "$CLEANUP_ALL" == true ]] && [[ -z "$CONFIG_FILE" ]]; then
if ! find_proxmox_config; then if ! find_proxmox_config; then
@ -4022,6 +4448,24 @@ main() {
return return
fi fi
# Handle lab lifecycle commands
if [[ "$LAB_DOWN" == true ]]; then
phase_lab_down
return
fi
if [[ "$LAB_UP" == true ]]; then
phase_lab_up
return
fi
# Handle --resources and --labs with config file
if [[ "$RESOURCES" == true ]] || [[ "$LIST_LABS" == true ]]; then
[[ "$RESOURCES" == true ]] && do_resources
[[ "$LIST_LABS" == true ]] && do_list_labs
return
fi
# Reconcile existing deployments (interactive menu on re-runs) # Reconcile existing deployments (interactive menu on re-runs)
if [[ "$PHASE" == "all" ]] && [[ "$DRY_RUN" != true ]]; then if [[ "$PHASE" == "all" ]] && [[ "$DRY_RUN" != true ]]; then
reconcile_existing reconcile_existing

94
incusos/incusos-seed
View File

@ -246,6 +246,45 @@ parse_args() {
done done
} }
# ---------------------------------------------------------------------------
# Platform detection
# ---------------------------------------------------------------------------
# ISO_TOOL is set by find_iso_tool() and used by create_iso_output()
ISO_TOOL=""
find_iso_tool() {
# Prefer genisoimage (Linux), fall back to mkisofs (macOS cdrtools)
if command -v genisoimage &>/dev/null; then
ISO_TOOL="genisoimage"
return 0
elif command -v mkisofs &>/dev/null; then
ISO_TOOL="mkisofs"
return 0
fi
return 1
}
# MKFS_FAT_TOOL is set by find_mkfs_fat_tool()
MKFS_FAT_TOOL=""
find_mkfs_fat_tool() {
if command -v mkfs.fat &>/dev/null; then
MKFS_FAT_TOOL="mkfs.fat"
return 0
elif [[ -x /usr/sbin/mkfs.fat ]]; then
MKFS_FAT_TOOL="/usr/sbin/mkfs.fat"
return 0
elif [[ -x /sbin/mkfs.fat ]]; then
MKFS_FAT_TOOL="/sbin/mkfs.fat"
return 0
elif command -v newfs_msdos &>/dev/null; then
MKFS_FAT_TOOL="newfs_msdos"
return 0
fi
return 1
}
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Validation # Validation
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@ -289,22 +328,27 @@ validate_args() {
fi fi
if [[ "$FORMAT" == "iso" ]]; then if [[ "$FORMAT" == "iso" ]]; then
if ! command -v genisoimage &>/dev/null; then if ! find_iso_tool; then
error "ISO 9660 image creation requires genisoimage but it was not found" error "ISO 9660 image creation requires genisoimage (Linux) or mkisofs (macOS)"
error "Install with: sudo apt install genisoimage" case "$(uname -s)" in
Darwin) error "Install with: brew install cdrtools" ;;
*) error "Install with: sudo apt install genisoimage" ;;
esac
exit 1 exit 1
fi fi
fi fi
if [[ "$FORMAT" == "fat" ]]; then if [[ "$FORMAT" == "fat" ]]; then
# mkfs.fat lives in /usr/sbin or /sbin which may not be in PATH for non-root users local has_mkfs=false
local has_mkfs_fat=false if find_mkfs_fat_tool; then
if command -v mkfs.fat &>/dev/null || [[ -x /usr/sbin/mkfs.fat ]] || [[ -x /sbin/mkfs.fat ]]; then has_mkfs=true
has_mkfs_fat=true
fi fi
if ! command -v mcopy &>/dev/null && [[ "$has_mkfs_fat" != true ]]; then if ! command -v mcopy &>/dev/null && [[ "$has_mkfs" != true ]]; then
error "FAT image creation requires mtools (mcopy) or dosfstools (mkfs.fat)" error "FAT image creation requires mtools (mcopy) or dosfstools (mkfs.fat)"
error "Install with: sudo apt install mtools dosfstools" case "$(uname -s)" in
Darwin) error "Install with: brew install mtools" ;;
*) error "Install with: sudo apt install mtools dosfstools" ;;
esac
exit 1 exit 1
fi fi
fi fi
@ -373,7 +417,8 @@ detect_client_cert() {
return return
fi fi
local incus_version local incus_version
incus_version=$(incus version 2>/dev/null | grep -oP 'Client version: \K[0-9.]+' || echo "unknown") incus_version=$(incus version 2>/dev/null | sed -n 's/.*Client version: \([0-9.]*\).*/\1/p' || echo "unknown")
[[ -z "$incus_version" ]] && incus_version="unknown"
detail "Incus CLI v${incus_version} found but no certificate available" detail "Incus CLI v${incus_version} found but no certificate available"
detail "Generate one with: incus remote list (triggers auto-generation)" detail "Generate one with: incus remote list (triggers auto-generation)"
fi fi
@ -418,7 +463,7 @@ generate_install_yaml() {
{ {
echo "# IncusOS installation configuration" echo "# IncusOS installation configuration"
echo "# Generated by ${SCRIPT_NAME} v${VERSION} on $(date -Iseconds)" echo "# Generated by ${SCRIPT_NAME} v${VERSION} on $(date '+%Y-%m-%dT%H:%M:%S%z')"
if [[ "$FORCE_INSTALL" == true ]]; then if [[ "$FORCE_INSTALL" == true ]]; then
echo "force_install: true" echo "force_install: true"
@ -612,7 +657,9 @@ create_tar_output() {
create_iso_output() { create_iso_output() {
step "Creating ISO 9660 image: ${OUTPUT}" step "Creating ISO 9660 image: ${OUTPUT}"
genisoimage -o "$OUTPUT" -V SEED_DATA -J -r "$WORKDIR"/ 2>/dev/null # ISO_TOOL was set by find_iso_tool() during validation
# Both genisoimage and mkisofs accept the same core flags
"$ISO_TOOL" -o "$OUTPUT" -V SEED_DATA -J -r "$WORKDIR"/ 2>/dev/null
local size local size
size=$(du -h "$OUTPUT" | cut -f1) size=$(du -h "$OUTPUT" | cut -f1)
@ -623,23 +670,19 @@ create_fat_output() {
step "Creating FAT image: ${OUTPUT} (${FAT_SIZE} MiB)" step "Creating FAT image: ${OUTPUT} (${FAT_SIZE} MiB)"
# Create empty image # Create empty image
dd if=/dev/zero of="$OUTPUT" bs=1M count="$FAT_SIZE" status=none dd if=/dev/zero of="$OUTPUT" bs=1048576 count="$FAT_SIZE" 2>/dev/null
# mkfs.fat lives in /usr/sbin or /sbin which may not be in PATH for non-root users # MKFS_FAT_TOOL was set by find_mkfs_fat_tool() during validation
local mkfs_fat="" if [[ -n "$MKFS_FAT_TOOL" ]]; then
if command -v mkfs.fat &>/dev/null; then if [[ "$MKFS_FAT_TOOL" == "newfs_msdos" ]]; then
mkfs_fat="mkfs.fat" # macOS built-in: -F 12 for FAT12, -v for volume label
elif [[ -x /usr/sbin/mkfs.fat ]]; then newfs_msdos -F 12 -v SEED_DATA "$OUTPUT" >/dev/null
mkfs_fat="/usr/sbin/mkfs.fat" else
elif [[ -x /sbin/mkfs.fat ]]; then # Linux mkfs.fat
mkfs_fat="/sbin/mkfs.fat" "$MKFS_FAT_TOOL" -n SEED_DATA "$OUTPUT" >/dev/null
fi fi
if [[ -n "$mkfs_fat" ]]; then
"$mkfs_fat" -n SEED_DATA "$OUTPUT" >/dev/null
if command -v mcopy &>/dev/null; then if command -v mcopy &>/dev/null; then
# Use mtools to copy files without mounting
for f in "${WORKDIR}"/*.yaml; do for f in "${WORKDIR}"/*.yaml; do
[[ -f "$f" ]] && mcopy -i "$OUTPUT" "$f" "::/$(basename "$f")" [[ -f "$f" ]] && mcopy -i "$OUTPUT" "$f" "::/$(basename "$f")"
done done
@ -647,7 +690,6 @@ create_fat_output() {
warn "mtools not installed -- FAT image created but files not copied" warn "mtools not installed -- FAT image created but files not copied"
warn "Install mtools and re-run, or manually mount and copy seed files" warn "Install mtools and re-run, or manually mount and copy seed files"
warn "Files are in: ${WORKDIR}/" warn "Files are in: ${WORKDIR}/"
# Keep workdir around
trap - EXIT trap - EXIT
return return
fi fi

806
incusos/lab-test
View File

@ -73,6 +73,13 @@ ${BOLD}OPTIONS${RESET}
${CYAN}cluster${RESET} -- form a 3-node cluster ${CYAN}cluster${RESET} -- form a 3-node cluster
${CYAN}workload${RESET} -- cluster workload tests ${CYAN}workload${RESET} -- cluster workload tests
${CYAN}migrate${RESET} -- migration and evacuation tests ${CYAN}migrate${RESET} -- migration and evacuation tests
${CYAN}storage${RESET} -- storage pool and volume operations
${CYAN}network${RESET} -- network features and ACLs
${CYAN}projects${RESET} -- project isolation and profiles
${CYAN}backup${RESET} -- backup, snapshot, and restore
${CYAN}limits${RESET} -- resource limits and QoS enforcement
${CYAN}security${RESET} -- container security and namespaces
${CYAN}resilience${RESET} -- HA, failover, and cluster resilience
${CYAN}all${RESET} -- run all phases (default) ${CYAN}all${RESET} -- run all phases (default)
${BOLD}--skip-deploy${RESET} Skip deploy/status check (assume VMs running) ${BOLD}--skip-deploy${RESET} Skip deploy/status check (assume VMs running)
${BOLD}--cleanup${RESET} Tear down test artifacts without destroying VMs ${BOLD}--cleanup${RESET} Tear down test artifacts without destroying VMs
@ -187,10 +194,10 @@ validate_args() {
fi fi
case "$PHASE" in case "$PHASE" in
deploy|single|cluster|workload|migrate|all) ;; deploy|single|cluster|workload|migrate|storage|network|projects|backup|limits|security|resilience|all) ;;
*) *)
error "Invalid phase: ${PHASE}" error "Invalid phase: ${PHASE}"
error "Supported: deploy, single, cluster, workload, migrate, all" error "Supported: deploy, single, cluster, workload, migrate, storage, network, projects, backup, limits, security, resilience, all"
exit 1 exit 1
;; ;;
esac esac
@ -950,6 +957,750 @@ do_cleanup() {
echo "" echo ""
} }
# ---------------------------------------------------------------------------
# Phase 6: Storage Operations
# ---------------------------------------------------------------------------
phase_storage() {
step "Phase 6: Storage Operations"
echo ""
local target="${VM_NAMES[0]}"
info "Using node: ${target}"
if [[ "$DRY_RUN" != true ]] && ! incus info "${target}:" &>/dev/null 2>&1; then
fail "Cannot connect to remote '${target}'"
return 1
fi
# Test 1: List storage pools
step "Test: List storage pools"
if [[ "$DRY_RUN" == true ]]; then
pass "Storage pool listing (dry-run)"
else
local pools
pools=$(incus storage list "${target}:" --format csv 2>/dev/null) || pools=""
if [[ -n "$pools" ]]; then
pass "Storage pool listing (found pools)"
detail "$pools"
else
fail "No storage pools found"
fi
fi
echo ""
# Test 2: Create custom storage volume
step "Test: Create and manage storage volume"
local vol_name="test-vol-$$"
if [[ "$DRY_RUN" == true ]]; then
detail "[dry-run] incus storage volume create ${target}:local ${vol_name}"
detail "[dry-run] incus storage volume show ${target}:local ${vol_name}"
detail "[dry-run] incus storage volume delete ${target}:local ${vol_name}"
pass "Storage volume lifecycle (dry-run)"
else
if incus storage volume create "${target}:local" "${vol_name}" 2>/dev/null; then
local vol_info
vol_info=$(incus storage volume show "${target}:local" "${vol_name}" 2>/dev/null) || vol_info=""
if [[ -n "$vol_info" ]]; then
pass "Storage volume create + show"
else
fail "Volume created but show failed"
fi
incus storage volume delete "${target}:local" "${vol_name}" 2>/dev/null || true
detail "Volume cleaned up"
else
fail "Storage volume create failed"
fi
fi
echo ""
# Test 3: Volume snapshot
step "Test: Volume snapshot lifecycle"
local snap_vol="test-snapvol-$$"
if [[ "$DRY_RUN" == true ]]; then
detail "[dry-run] create volume, snapshot, restore, delete"
pass "Volume snapshot lifecycle (dry-run)"
else
if incus storage volume create "${target}:local" "${snap_vol}" 2>/dev/null; then
if incus storage volume snapshot create "${target}:local" "${snap_vol}" test-snap 2>/dev/null; then
pass "Volume snapshot create"
# Restore
if incus storage volume snapshot restore "${target}:local" "${snap_vol}" test-snap 2>/dev/null; then
pass "Volume snapshot restore"
else
fail "Volume snapshot restore failed"
fi
else
fail "Volume snapshot create failed"
fi
incus storage volume delete "${target}:local" "${snap_vol}" 2>/dev/null || true
detail "Snapshot volume cleaned up"
else
fail "Storage volume create failed (for snapshot test)"
fi
fi
echo ""
# Test 4: Pool usage reporting
step "Test: Storage pool usage"
if [[ "$DRY_RUN" == true ]]; then
pass "Pool usage reporting (dry-run)"
else
local usage
usage=$(incus storage info "${target}:local" 2>/dev/null) || usage=""
if [[ -n "$usage" ]]; then
pass "Storage pool usage reporting"
detail "$(echo "$usage" | head -5)"
else
fail "Could not get pool usage info"
fi
fi
echo ""
}
# ---------------------------------------------------------------------------
# Phase 7: Network Features
# ---------------------------------------------------------------------------
phase_network() {
step "Phase 7: Network Features"
echo ""
local target="${VM_NAMES[0]}"
info "Using node: ${target}"
if [[ "$DRY_RUN" != true ]] && ! incus info "${target}:" &>/dev/null 2>&1; then
fail "Cannot connect to remote '${target}'"
return 1
fi
# Test 1: List networks
step "Test: List networks"
if [[ "$DRY_RUN" == true ]]; then
pass "Network listing (dry-run)"
else
local nets
nets=$(incus network list "${target}:" --format csv 2>/dev/null) || nets=""
if [[ -n "$nets" ]]; then
pass "Network listing"
detail "$nets"
else
fail "No networks found"
fi
fi
echo ""
# Test 2: Create and delete a managed network bridge
step "Test: Create managed network bridge"
local net_name="test-br-$$"
if [[ "$DRY_RUN" == true ]]; then
detail "[dry-run] incus network create ${target}:${net_name}"
detail "[dry-run] incus network delete ${target}:${net_name}"
pass "Network bridge lifecycle (dry-run)"
else
if incus network create "${target}:${net_name}" \
ipv4.address=10.99.99.1/24 ipv4.nat=true ipv6.address=none 2>/dev/null; then
local net_info
net_info=$(incus network show "${target}:${net_name}" 2>/dev/null) || net_info=""
if [[ -n "$net_info" ]]; then
pass "Network bridge create + show"
else
fail "Network created but show failed"
fi
incus network delete "${target}:${net_name}" 2>/dev/null || true
detail "Network cleaned up"
else
fail "Network bridge create failed"
fi
fi
echo ""
# Test 3: Network ACL
step "Test: Network ACL create/delete"
local acl_name="test-acl-$$"
if [[ "$DRY_RUN" == true ]]; then
detail "[dry-run] incus network acl create/delete"
pass "Network ACL lifecycle (dry-run)"
else
if incus network acl create "${target}:${acl_name}" 2>/dev/null; then
pass "Network ACL create"
incus network acl delete "${target}:${acl_name}" 2>/dev/null || true
detail "ACL cleaned up"
else
skip "Network ACL not supported (may require OVN)"
fi
fi
echo ""
# Test 4: Network forward (port forwarding)
step "Test: Network forward"
if [[ "$DRY_RUN" == true ]]; then
detail "[dry-run] incus network forward create/delete"
pass "Network forward (dry-run)"
else
# Create a temporary instance to test forwarding
local fw_ct="test-fw-$$"
if incus launch images:debian/12 "${target}:${fw_ct}" 2>/dev/null; then
sleep 5
local ct_ip
ct_ip=$(incus list "${target}:${fw_ct}" --format csv -c 4 2>/dev/null | cut -d' ' -f1) || ct_ip=""
if [[ -n "$ct_ip" ]] && [[ "$ct_ip" != *"()"* ]]; then
if incus network forward create "${target}:incusbr0" 10.66.66.100 2>/dev/null; then
pass "Network forward create"
incus network forward delete "${target}:incusbr0" 10.66.66.100 2>/dev/null || true
else
skip "Network forward not supported on this network type"
fi
else
skip "Could not get container IP for forward test"
fi
incus delete "${target}:${fw_ct}" --force 2>/dev/null || true
else
skip "Could not create container for forward test"
fi
fi
echo ""
}
# ---------------------------------------------------------------------------
# Phase 8: Projects & Profiles
# ---------------------------------------------------------------------------
phase_projects() {
step "Phase 8: Projects & Profiles"
echo ""
local target="${VM_NAMES[0]}"
info "Using node: ${target}"
if [[ "$DRY_RUN" != true ]] && ! incus info "${target}:" &>/dev/null 2>&1; then
fail "Cannot connect to remote '${target}'"
return 1
fi
# Test 1: Create project with limits
step "Test: Create project with resource limits"
local proj_name="test-proj-$$"
if [[ "$DRY_RUN" == true ]]; then
detail "[dry-run] incus project create/delete"
pass "Project lifecycle (dry-run)"
else
if incus project create "${target}:${proj_name}" \
-c features.images=false \
-c features.profiles=true \
-c limits.instances=5 2>/dev/null; then
local proj_info
proj_info=$(incus project show "${target}:${proj_name}" 2>/dev/null) || proj_info=""
if [[ -n "$proj_info" ]]; then
pass "Project create with limits"
else
fail "Project created but show failed"
fi
incus project delete "${target}:${proj_name}" 2>/dev/null || true
detail "Project cleaned up"
else
fail "Project create failed"
fi
fi
echo ""
# Test 2: Custom profile
step "Test: Create and apply custom profile"
local profile_name="test-profile-$$"
if [[ "$DRY_RUN" == true ]]; then
detail "[dry-run] incus profile create/delete"
pass "Profile lifecycle (dry-run)"
else
if incus profile create "${target}:${profile_name}" 2>/dev/null; then
# Set some config on the profile
incus profile set "${target}:${profile_name}" limits.cpu=1 limits.memory=256MiB 2>/dev/null || true
local prof_info
prof_info=$(incus profile show "${target}:${profile_name}" 2>/dev/null) || prof_info=""
if echo "$prof_info" | grep -q "limits.cpu"; then
pass "Profile create + configure"
else
fail "Profile created but config not applied"
fi
incus profile delete "${target}:${profile_name}" 2>/dev/null || true
detail "Profile cleaned up"
else
fail "Profile create failed"
fi
fi
echo ""
# Test 3: Profile inheritance
step "Test: Launch container with custom profile"
local ct_name="test-prof-ct-$$"
local prof2="test-prof2-$$"
if [[ "$DRY_RUN" == true ]]; then
detail "[dry-run] create profile, launch container with profile, verify config"
pass "Profile inheritance (dry-run)"
else
if incus profile create "${target}:${prof2}" 2>/dev/null; then
incus profile set "${target}:${prof2}" limits.cpu=1 2>/dev/null || true
if incus launch images:debian/12 "${target}:${ct_name}" --profile default --profile "${prof2}" 2>/dev/null; then
sleep 5
local ct_config
ct_config=$(incus config show "${target}:${ct_name}" 2>/dev/null) || ct_config=""
if echo "$ct_config" | grep -q "limits.cpu"; then
pass "Container inherits profile config"
else
# Profile config may be in expanded config
local expanded
expanded=$(incus config show "${target}:${ct_name}" --expanded 2>/dev/null) || expanded=""
if echo "$expanded" | grep -q "limits.cpu"; then
pass "Container inherits profile config (expanded)"
else
fail "Profile config not inherited"
fi
fi
incus delete "${target}:${ct_name}" --force 2>/dev/null || true
else
fail "Container launch with profile failed"
fi
incus profile delete "${target}:${prof2}" 2>/dev/null || true
detail "Cleaned up"
else
fail "Profile create failed"
fi
fi
echo ""
}
# ---------------------------------------------------------------------------
# Phase 9: Backup & Restore
# ---------------------------------------------------------------------------
phase_backup() {
step "Phase 9: Backup & Restore"
echo ""
local target="${VM_NAMES[0]}"
info "Using node: ${target}"
if [[ "$DRY_RUN" != true ]] && ! incus info "${target}:" &>/dev/null 2>&1; then
fail "Cannot connect to remote '${target}'"
return 1
fi
# Test 1: Instance snapshot lifecycle
step "Test: Instance snapshot create/restore/delete"
local ct_name="test-snap-$$"
if [[ "$DRY_RUN" == true ]]; then
detail "[dry-run] create instance, snapshot, restore, delete"
pass "Instance snapshot lifecycle (dry-run)"
else
if incus launch images:debian/12 "${target}:${ct_name}" 2>/dev/null; then
sleep 5
# Create a marker file before snapshot
incus exec "${target}:${ct_name}" -- touch /tmp/before-snap 2>/dev/null || true
# Create snapshot
if incus snapshot create "${target}:${ct_name}" test-snap1 2>/dev/null; then
pass "Instance snapshot create"
# Modify instance after snapshot
incus exec "${target}:${ct_name}" -- touch /tmp/after-snap 2>/dev/null || true
# Restore snapshot
incus stop "${target}:${ct_name}" --force 2>/dev/null || true
if incus snapshot restore "${target}:${ct_name}" test-snap1 2>/dev/null; then
incus start "${target}:${ct_name}" 2>/dev/null || true
sleep 3
# Verify marker file exists (from before snapshot)
if incus exec "${target}:${ct_name}" -- test -f /tmp/before-snap 2>/dev/null; then
pass "Instance snapshot restore (marker file preserved)"
else
fail "Snapshot restored but marker file missing"
fi
# Verify after-snap file is gone (rolled back)
if ! incus exec "${target}:${ct_name}" -- test -f /tmp/after-snap 2>/dev/null; then
pass "Snapshot rollback (post-snapshot changes reverted)"
else
fail "Post-snapshot file still exists after restore"
fi
else
fail "Instance snapshot restore failed"
fi
# Delete snapshot
if incus snapshot delete "${target}:${ct_name}" test-snap1 2>/dev/null; then
pass "Instance snapshot delete"
else
fail "Instance snapshot delete failed"
fi
else
fail "Instance snapshot create failed"
fi
incus delete "${target}:${ct_name}" --force 2>/dev/null || true
detail "Cleaned up"
else
fail "Container launch failed (for snapshot test)"
fi
fi
echo ""
# Test 2: Export/import round-trip
step "Test: Export and import instance"
local exp_ct="test-export-$$"
if [[ "$DRY_RUN" == true ]]; then
detail "[dry-run] export/import container"
pass "Export/import round-trip (dry-run)"
else
if incus launch images:debian/12 "${target}:${exp_ct}" 2>/dev/null; then
sleep 5
incus exec "${target}:${exp_ct}" -- sh -c "echo test-data > /tmp/export-marker" 2>/dev/null || true
local export_file="/tmp/test-export-$$.tar.gz"
incus stop "${target}:${exp_ct}" --force 2>/dev/null || true
if incus export "${target}:${exp_ct}" "$export_file" 2>/dev/null; then
pass "Instance export"
incus delete "${target}:${exp_ct}" --force 2>/dev/null || true
# Import
if incus import "${target}:" "$export_file" 2>/dev/null; then
incus start "${target}:${exp_ct}" 2>/dev/null || true
sleep 5
local marker
marker=$(incus exec "${target}:${exp_ct}" -- cat /tmp/export-marker 2>/dev/null) || marker=""
if [[ "$marker" == "test-data" ]]; then
pass "Instance import (data preserved)"
else
fail "Instance imported but data not preserved"
fi
else
fail "Instance import failed"
fi
else
fail "Instance export failed"
fi
incus delete "${target}:${exp_ct}" --force 2>/dev/null || true
rm -f "$export_file"
detail "Cleaned up"
else
fail "Container launch failed (for export test)"
fi
fi
echo ""
}
# ---------------------------------------------------------------------------
# Phase 10: Resource Limits
# ---------------------------------------------------------------------------
phase_limits() {
step "Phase 10: Resource Limits & QoS"
echo ""
local target="${VM_NAMES[0]}"
info "Using node: ${target}"
if [[ "$DRY_RUN" != true ]] && ! incus info "${target}:" &>/dev/null 2>&1; then
fail "Cannot connect to remote '${target}'"
return 1
fi
# Test 1: CPU limit
step "Test: CPU limit enforcement"
local ct_name="test-limits-$$"
if [[ "$DRY_RUN" == true ]]; then
detail "[dry-run] launch container with CPU/memory limits"
pass "CPU limit (dry-run)"
else
if incus launch images:debian/12 "${target}:${ct_name}" \
-c limits.cpu=1 -c limits.memory=256MiB 2>/dev/null; then
sleep 5
# Verify limits are applied
local cpus
cpus=$(incus exec "${target}:${ct_name}" -- nproc 2>/dev/null) || cpus=""
if [[ "$cpus" == "1" ]]; then
pass "CPU limit enforced (nproc=${cpus})"
else
fail "CPU limit not enforced (nproc=${cpus}, expected 1)"
fi
# Test 2: Memory limit
step "Test: Memory limit enforcement"
local mem_limit
mem_limit=$(incus exec "${target}:${ct_name}" -- sh -c \
"cat /sys/fs/cgroup/memory.max 2>/dev/null || cat /sys/fs/cgroup/memory/memory.limit_in_bytes 2>/dev/null || echo unknown" 2>/dev/null) || mem_limit="unknown"
if [[ "$mem_limit" != "unknown" ]] && [[ "$mem_limit" != "max" ]]; then
local mem_mib=$((mem_limit / 1024 / 1024))
if [[ $mem_mib -le 260 ]] && [[ $mem_mib -ge 240 ]]; then
pass "Memory limit enforced (~${mem_mib}MiB)"
else
fail "Memory limit unexpected: ${mem_mib}MiB (expected ~256)"
fi
else
skip "Could not read memory limit (cgroup v1/v2 difference)"
fi
incus delete "${target}:${ct_name}" --force 2>/dev/null || true
detail "Cleaned up"
else
fail "Container launch with limits failed"
fi
fi
echo ""
# Test 3: Disk I/O limits
step "Test: Disk I/O limit configuration"
local io_ct="test-io-$$"
if [[ "$DRY_RUN" == true ]]; then
detail "[dry-run] set disk I/O limits"
pass "Disk I/O limits (dry-run)"
else
if incus launch images:debian/12 "${target}:${io_ct}" 2>/dev/null; then
sleep 5
# Set I/O limits on root disk
if incus config device set "${target}:${io_ct}" root limits.read=10MB limits.write=10MB 2>/dev/null; then
pass "Disk I/O limit configuration accepted"
else
skip "Disk I/O limits not supported on this storage driver"
fi
incus delete "${target}:${io_ct}" --force 2>/dev/null || true
detail "Cleaned up"
else
fail "Container launch failed (for I/O test)"
fi
fi
echo ""
}
# ---------------------------------------------------------------------------
# Phase 11: Security
# ---------------------------------------------------------------------------
phase_security() {
step "Phase 11: Security"
echo ""
local target="${VM_NAMES[0]}"
info "Using node: ${target}"
if [[ "$DRY_RUN" != true ]] && ! incus info "${target}:" &>/dev/null 2>&1; then
fail "Cannot connect to remote '${target}'"
return 1
fi
# Test 1: Unprivileged container (default)
step "Test: Unprivileged container (default)"
local ct_name="test-sec-$$"
if [[ "$DRY_RUN" == true ]]; then
detail "[dry-run] verify container is unprivileged"
pass "Unprivileged container (dry-run)"
else
if incus launch images:debian/12 "${target}:${ct_name}" 2>/dev/null; then
sleep 5
# Check that the container is running unprivileged (uid mapping)
local uid
uid=$(incus exec "${target}:${ct_name}" -- id -u 2>/dev/null) || uid=""
local config
config=$(incus config show "${target}:${ct_name}" 2>/dev/null) || config=""
if echo "$config" | grep -q "security.privileged: \"true\""; then
fail "Container is privileged (unexpected default)"
else
pass "Container is unprivileged (default)"
fi
incus delete "${target}:${ct_name}" --force 2>/dev/null || true
else
fail "Container launch failed"
fi
fi
echo ""
# Test 2: Privileged container
step "Test: Privileged container"
local priv_ct="test-priv-$$"
if [[ "$DRY_RUN" == true ]]; then
detail "[dry-run] launch privileged container"
pass "Privileged container (dry-run)"
else
if incus launch images:debian/12 "${target}:${priv_ct}" \
-c security.privileged=true 2>/dev/null; then
sleep 5
local config
config=$(incus config show "${target}:${priv_ct}" 2>/dev/null) || config=""
if echo "$config" | grep -q "security.privileged"; then
pass "Privileged container created"
else
fail "Privileged flag not set"
fi
incus delete "${target}:${priv_ct}" --force 2>/dev/null || true
detail "Cleaned up"
else
fail "Privileged container launch failed"
fi
fi
echo ""
# Test 3: Container nesting
step "Test: Container nesting support"
local nest_ct="test-nest-$$"
if [[ "$DRY_RUN" == true ]]; then
detail "[dry-run] launch container with nesting enabled"
pass "Container nesting (dry-run)"
else
if incus launch images:debian/12 "${target}:${nest_ct}" \
-c security.nesting=true 2>/dev/null; then
sleep 5
local config
config=$(incus config show "${target}:${nest_ct}" 2>/dev/null) || config=""
if echo "$config" | grep -q "security.nesting"; then
pass "Nesting support enabled"
else
fail "Nesting flag not set"
fi
incus delete "${target}:${nest_ct}" --force 2>/dev/null || true
detail "Cleaned up"
else
fail "Container with nesting launch failed"
fi
fi
echo ""
# Test 4: ID mapping (user namespace isolation)
step "Test: User namespace isolation"
local idmap_ct="test-idmap-$$"
if [[ "$DRY_RUN" == true ]]; then
detail "[dry-run] verify UID/GID mapping in container"
pass "User namespace isolation (dry-run)"
else
if incus launch images:debian/12 "${target}:${idmap_ct}" 2>/dev/null; then
sleep 5
# In unprivileged containers, root inside maps to a high UID outside
local inner_uid outer_pid
inner_uid=$(incus exec "${target}:${idmap_ct}" -- id -u 2>/dev/null) || inner_uid=""
if [[ "$inner_uid" == "0" ]]; then
# Root inside the container maps to a high UID on the host
pass "User namespace isolation (root maps to uid 0 inside container)"
else
fail "Unexpected uid inside container: ${inner_uid}"
fi
incus delete "${target}:${idmap_ct}" --force 2>/dev/null || true
detail "Cleaned up"
else
fail "Container launch failed (for idmap test)"
fi
fi
echo ""
}
# ---------------------------------------------------------------------------
# Phase 12: HA & Cluster Resilience
# ---------------------------------------------------------------------------
phase_resilience() {
step "Phase 12: HA & Cluster Resilience"
echo ""
local count=${#VM_NAMES[@]}
if [[ $count -lt 3 ]]; then
skip "Resilience tests require at least 3 nodes (have ${count})"
return
fi
local cluster_remote="${VM_NAMES[0]}"
info "Using cluster via: ${cluster_remote}"
if [[ "$DRY_RUN" != true ]] && ! incus info "${cluster_remote}:" &>/dev/null 2>&1; then
fail "Cannot connect to remote '${cluster_remote}'"
return 1
fi
# Verify we're in a cluster
if [[ "$DRY_RUN" != true ]]; then
local member_count
member_count=$(incus cluster list "${cluster_remote}:" --format csv 2>/dev/null | wc -l) || member_count=0
if [[ $member_count -lt 2 ]]; then
skip "Not in a cluster (${member_count} members). Run cluster phase first."
return
fi
info "Cluster has ${member_count} members"
fi
echo ""
# Test 1: Cluster member listing and health
step "Test: Cluster health check"
if [[ "$DRY_RUN" == true ]]; then
detail "[dry-run] incus cluster list"
pass "Cluster health (dry-run)"
else
local members
members=$(incus cluster list "${cluster_remote}:" --format csv -c ns 2>/dev/null) || members=""
local all_online=true
while IFS=',' read -r name status; do
[[ -z "$name" ]] && continue
if [[ "$status" != "Online" ]] && [[ "$status" != "ONLINE" ]]; then
fail "Member ${name} is ${status}"
all_online=false
fi
done <<< "$members"
if [[ "$all_online" == true ]]; then
pass "All cluster members online"
fi
fi
echo ""
# Test 2: Database leader identification
step "Test: Database leader identification"
if [[ "$DRY_RUN" == true ]]; then
detail "[dry-run] incus cluster list (check roles)"
pass "Database leader (dry-run)"
else
local leader
leader=$(incus cluster list "${cluster_remote}:" --format csv -c nR 2>/dev/null | grep -i "database-leader" | head -1) || leader=""
if [[ -n "$leader" ]]; then
pass "Database leader identified: $(echo "$leader" | cut -d',' -f1)"
else
# Older Incus may not show roles in CSV
skip "Could not identify database leader (format may differ)"
fi
fi
echo ""
# Test 3: Workload survives member reboot (via Proxmox stop/start)
step "Test: Workload availability during member restart"
if [[ "$DRY_RUN" == true ]]; then
detail "[dry-run] Would test workload survival through member restart"
detail "[dry-run] This test requires Proxmox access and is not safe in dry-run"
skip "Workload survival test (dry-run -- requires live infrastructure)"
else
# This test just verifies that the cluster can handle member state queries
# Actual stop/start requires Proxmox and is too destructive for automated tests
local member_list
member_list=$(incus cluster list "${cluster_remote}:" --format csv -c n 2>/dev/null) || member_list=""
if [[ -n "$member_list" ]]; then
pass "Cluster member state queryable (stop/start test requires manual execution)"
detail "To test: stop a non-leader node via Proxmox, verify cluster still works"
else
fail "Cannot query cluster members"
fi
fi
echo ""
# Test 4: Cluster warnings
step "Test: Cluster warnings check"
if [[ "$DRY_RUN" == true ]]; then
detail "[dry-run] incus warning list"
pass "Cluster warnings (dry-run)"
else
local warnings
warnings=$(incus warning list "${cluster_remote}:" --format csv 2>/dev/null) || warnings=""
if [[ -z "$warnings" ]]; then
pass "No cluster warnings"
else
local warn_count
warn_count=$(echo "$warnings" | wc -l)
warn "Cluster has ${warn_count} warning(s):"
detail "$warnings"
pass "Cluster warnings retrieved (${warn_count} active)"
fi
fi
echo ""
}
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Summary # Summary
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@ -1003,35 +1754,62 @@ main() {
phase_deploy phase_deploy
;; ;;
single) single)
if [[ "$SKIP_DEPLOY" != true ]]; then if [[ "$SKIP_DEPLOY" != true ]]; then phase_deploy; fi
phase_deploy
fi
phase_single phase_single
;; ;;
cluster) cluster)
if [[ "$SKIP_DEPLOY" != true ]]; then if [[ "$SKIP_DEPLOY" != true ]]; then phase_deploy; fi
phase_deploy
fi
phase_cluster phase_cluster
;; ;;
workload) workload)
if [[ "$SKIP_DEPLOY" != true ]]; then if [[ "$SKIP_DEPLOY" != true ]]; then phase_deploy; fi
phase_deploy
fi
phase_workload phase_workload
;; ;;
migrate) migrate)
if [[ "$SKIP_DEPLOY" != true ]]; then if [[ "$SKIP_DEPLOY" != true ]]; then phase_deploy; fi
phase_deploy
fi
phase_migrate phase_migrate
;; ;;
storage)
if [[ "$SKIP_DEPLOY" != true ]]; then phase_deploy; fi
phase_storage
;;
network)
if [[ "$SKIP_DEPLOY" != true ]]; then phase_deploy; fi
phase_network
;;
projects)
if [[ "$SKIP_DEPLOY" != true ]]; then phase_deploy; fi
phase_projects
;;
backup)
if [[ "$SKIP_DEPLOY" != true ]]; then phase_deploy; fi
phase_backup
;;
limits)
if [[ "$SKIP_DEPLOY" != true ]]; then phase_deploy; fi
phase_limits
;;
security)
if [[ "$SKIP_DEPLOY" != true ]]; then phase_deploy; fi
phase_security
;;
resilience)
if [[ "$SKIP_DEPLOY" != true ]]; then phase_deploy; fi
phase_resilience
;;
all) all)
phase_deploy phase_deploy
phase_single phase_single
phase_cluster phase_cluster
phase_workload phase_workload
phase_migrate phase_migrate
phase_storage
phase_network
phase_projects
phase_backup
phase_limits
phase_security
phase_resilience
;; ;;
esac esac

324
notes/migration-guide.md Normal file
View File

@ -0,0 +1,324 @@
# Migration Guide: Importing Workloads into Incus
This guide covers migrating virtual machines and containers from other
hypervisors into Incus. Each section documents a specific migration path
with tested procedures where available.
All commands tested with Incus 6.21 on IncusOS.
---
## Overview
Incus provides several ways to import workloads:
| Tool | Use case |
|------|----------|
| `incus-migrate` | Official migration tool. Imports disk images, running instances, or physical machines |
| `incus import` | Import from an Incus backup file (`.tar.gz`) |
| `incus storage volume import` | Import a raw/qcow2 disk as a storage volume |
| `qemu-img convert` | Convert between disk formats (vmdk, qcow2, raw, vdi) |
### General migration workflow
1. **Export** the VM disk from the source hypervisor
2. **Convert** the disk to raw or qcow2 format (if needed)
3. **Import** into Incus via `incus-migrate` or manual volume import
4. **Configure** the Incus instance (network, boot, etc.)
5. **Verify** the migrated workload boots and runs correctly
---
## Proxmox VE to Incus
**Status**: documented, can be tested with lab VMs.
### Method 1: Using qemu-img (recommended)
```bash
# 1. Identify the disk on Proxmox
ssh root@proxmox pvesm list local-lvm | grep vm-<VMID>
# 2. Export the VM disk as raw
ssh root@proxmox qm disk export <VMID> scsi0 /tmp/disk.raw --format raw
# 3. Copy to the Incus host
scp root@proxmox:/tmp/disk.raw /tmp/disk.raw
# 4. Import as a storage volume
incus storage volume import <remote>:local /tmp/disk.raw migrated-disk --type=block
# 5. Create an empty VM
incus init --empty --vm migrated-vm <remote>:
# 6. Attach the imported disk
incus config device add <remote>:migrated-vm root disk pool=local source=migrated-disk
# 7. Configure for boot
incus config set <remote>:migrated-vm security.secureboot=false # if source lacks SB
incus start <remote>:migrated-vm
```
### Method 2: Using incus-migrate
```bash
# On the Incus host (or any machine with access to both)
incus-migrate
# Interactive prompts:
# 1. Select "Import from disk image"
# 2. Point to the exported raw/qcow2 file
# 3. Select target Incus server and storage pool
# 4. Choose VM instance type
```
### Notes
- Proxmox ZFS volumes can be exported with `zfs send` for faster transfer
- LVM-thin volumes: use `qm disk export` (abstracts the LVM details)
- UEFI VMs: ensure the Incus VM is configured with UEFI boot
(`security.secureboot=false` if the source OS lacks Secure Boot support)
- Network: the VM will get a new MAC address in Incus. Update any static
network config inside the VM after migration.
- Guest agent: install `incus-agent` inside the VM after migration for full
management capabilities.
---
## UTM to Incus
**Status**: documented (theoretical). Testable when running on macOS.
UTM stores VMs as `.utm` bundles (directories) containing qcow2 disk images.
### Procedure
```bash
# 1. Locate the UTM VM bundle
ls ~/Library/Containers/com.utmapp.UTM/Data/Documents/*.utm/
# 2. Find the qcow2 disk inside the bundle
ls <vm-name>.utm/Data/*.qcow2
# 3. Convert to raw (if needed)
qemu-img convert -f qcow2 -O raw <vm>.utm/Data/disk.qcow2 /tmp/disk.raw
# 4. Transfer to Incus host (if remote)
scp /tmp/disk.raw user@incus-host:/tmp/
# 5. Import into Incus (same as Proxmox method)
incus storage volume import <remote>:local /tmp/disk.raw migrated-disk --type=block
incus init --empty --vm migrated-vm <remote>:
incus config device add <remote>:migrated-vm root disk pool=local source=migrated-disk
incus start <remote>:migrated-vm
```
### Notes
- UTM supports both Apple Virtualization (arm64 only) and QEMU backends.
VMs from the QEMU backend are most compatible with Incus.
- Apple Virtualization VMs use a different format and may not be directly
importable.
- arm64 VMs from UTM can be imported into arm64 Incus hosts.
- x86_64 VMs (via QEMU backend on Intel Macs) can be imported into x86_64
Incus hosts.
---
## VMware Fusion to Incus
**Status**: documented (theoretical). Testable when VMware Fusion is available.
VMware Fusion stores VMs as `.vmwarevm` bundles containing `.vmdk` disk files.
### Procedure
```bash
# 1. Locate the VM bundle
ls ~/Virtual\ Machines.localized/*.vmwarevm/
# 2. Find the VMDK disk
ls <vm-name>.vmwarevm/*.vmdk
# 3. Convert VMDK to raw
qemu-img convert -f vmdk -O raw <vm>.vmwarevm/disk.vmdk /tmp/disk.raw
# 4. Transfer and import into Incus (same workflow)
scp /tmp/disk.raw user@incus-host:/tmp/
incus storage volume import <remote>:local /tmp/disk.raw migrated-disk --type=block
incus init --empty --vm migrated-vm <remote>:
incus config device add <remote>:migrated-vm root disk pool=local source=migrated-disk
incus start <remote>:migrated-vm
```
### OVA/OVF export (alternative)
```bash
# VMware Fusion can export VMs as OVA
# File -> Export to OVF...
# Extract VMDK from OVA (OVA is a tar archive)
tar xf machine.ova
ls *.vmdk
# Convert and import as above
qemu-img convert -f vmdk -O raw *.vmdk /tmp/disk.raw
```
### Notes
- Split VMDK files (multiple `.vmdk` files): use `qemu-img` which handles
split VMDK chains automatically.
- VMware Tools: uninstall before migration if possible, or install
`open-vm-tools` equivalent in the guest after migration.
- Network adapter type changes from vmxnet3 to virtio.
---
## Physical Machine to Incus
**Status**: documented. `incus-migrate` supports direct physical-to-virtual.
### Method 1: incus-migrate (recommended)
Run `incus-migrate` directly on the physical machine:
```bash
# On the physical machine (requires root)
sudo incus-migrate
# Interactive prompts guide you through:
# 1. Connect to Incus server
# 2. Select "Migrate this machine"
# 3. Choose instance type (container or VM)
# 4. incus-migrate streams the filesystem/disk to Incus
```
### Method 2: Disk image capture
```bash
# 1. Create a disk image from the physical machine
sudo dd if=/dev/sda of=/tmp/disk.raw bs=4M status=progress
# 2. Transfer to Incus host
scp /tmp/disk.raw user@incus-host:/tmp/
# 3. Import as usual
incus storage volume import <remote>:local /tmp/disk.raw migrated-disk --type=block
incus init --empty --vm migrated-vm <remote>:
incus config device add <remote>:migrated-vm root disk pool=local source=migrated-disk
```
### Notes
- `incus-migrate` is the cleanest approach for physical machines.
- Disk image capture preserves everything (bootloader, partitions) but
creates very large files. Use compression: `dd | gzip > disk.raw.gz`
- Shrink the disk image after transfer: `qemu-img convert -O raw disk.raw
disk-shrunk.raw` (removes trailing zeros).
- Hardware drivers: the guest may need to regenerate initramfs for the new
virtual hardware (virtio drivers).
---
## Container Migration (Docker/Podman to Incus)
**Status**: documented. Container filesystem import is straightforward.
### Docker to Incus container
```bash
# 1. Export the container filesystem
docker export <container-id> > /tmp/container-fs.tar
# 2. Import into Incus as a container
incus import <remote>: /tmp/container-fs.tar imported-container
# 3. Start
incus start <remote>:imported-container
```
### Podman to Incus container
```bash
# Same approach -- podman export is compatible
podman export <container-id> > /tmp/container-fs.tar
incus import <remote>: /tmp/container-fs.tar imported-container
```
### Docker image to Incus
```bash
# Save image as tar
docker save <image-name> > /tmp/image.tar
# Note: docker save creates an OCI image archive, not a filesystem tar.
# incus import expects a filesystem tar. Use docker export (from a running
# container) instead.
```
### Notes
- Docker containers are typically single-process. Incus containers run a
full init system. The imported container may need an init process
(systemd, openrc) installed.
- Networking: Docker networking doesn't translate. The imported container
gets Incus networking (bridge-based).
- Volumes: Docker volumes are NOT exported with `docker export`. Copy
volume data separately.
- For complex Docker Compose stacks, consider running Docker inside an Incus
VM instead of migrating individual containers.
---
## Disk Format Reference
| Format | Extension | Tools | Notes |
|--------|-----------|-------|-------|
| Raw | `.raw`, `.img` | `dd`, `qemu-img` | No overhead, largest size |
| QCOW2 | `.qcow2` | `qemu-img` | Sparse, snapshots, common in KVM/QEMU |
| VMDK | `.vmdk` | `qemu-img` | VMware format, may be split across files |
| VDI | `.vdi` | `qemu-img` | VirtualBox format |
| VHD/VHDX | `.vhd`, `.vhdx` | `qemu-img` | Hyper-V format |
### Converting between formats
```bash
# QCOW2 to raw
qemu-img convert -f qcow2 -O raw input.qcow2 output.raw
# VMDK to raw
qemu-img convert -f vmdk -O raw input.vmdk output.raw
# Raw to QCOW2 (for storage efficiency)
qemu-img convert -f raw -O qcow2 input.raw output.qcow2
# VDI to raw
qemu-img convert -f vdi -O raw input.vdi output.raw
# Check image info
qemu-img info disk.qcow2
```
---
## Post-Migration Checklist
After importing a workload into Incus:
- [ ] Verify the instance boots and runs correctly
- [ ] Update network configuration (new MAC address, new IP range)
- [ ] Install `incus-agent` inside VMs for full management
- [ ] Remove source-hypervisor-specific tools (VMware Tools, etc.)
- [ ] Regenerate initramfs if hardware changed significantly
- [ ] Update `/etc/fstab` if disk device names changed
- [ ] Test application functionality
- [ ] Set up backups in Incus (`incus export` or snapshots)
---
## References
- [incus-migrate documentation](https://linuxcontainers.org/incus/docs/main/howto/server_migrate/)
- [qemu-img manual](https://www.qemu.org/docs/master/tools/qemu-img.html)
- [Incus import/export](https://linuxcontainers.org/incus/docs/main/howto/instances_backup/)

46
notes/operations-center-guide.md
View File

@ -408,11 +408,13 @@ operations-center provisioning server system poweroff <name>
``` ```
**WARNING -- OC reboot breaks IncusOS on Proxmox**: the `system reboot` **WARNING -- OC reboot breaks IncusOS on Proxmox**: the `system reboot`
command sends a guest-level (ACPI) reboot. On Proxmox VMs with TPM2-based command sends a guest-level (ACPI) reboot. On Proxmox VMs, this likely
disk encryption, this corrupts the TPM PCR measurements. The IncusOS node corrupts the dqlite/Incus database through unclean shutdown. The IncusOS
becomes pingable but all services fail to start (port 8443 never opens). node becomes pingable (TPM disk decryption succeeds, OS boots) but the
Even a Proxmox-level stop/start does not recover the node. The only fix is Incus daemon never starts (port 8443 never opens). Even a Proxmox-level
to destroy and redeploy the VM. stop/start does not recover the node. The only fix is to destroy and
redeploy the VM. **Note**: this is an open research item -- the exact
failure mechanism is unconfirmed.
**Safe reboot alternative**: use Proxmox stop/start instead of OC reboot. **Safe reboot alternative**: use Proxmox stop/start instead of OC reboot.
The proper lifecycle for maintenance is: The proper lifecycle for maintenance is:
@ -768,7 +770,8 @@ ip neigh show | grep -i "$MAC" | awk '{print $1}'
EVACUATED or OFFLINE nodes EVACUATED or OFFLINE nodes
- **Stale server entries**: out-of-band cluster changes (via `incus` CLI) - **Stale server entries**: out-of-band cluster changes (via `incus` CLI)
create stale entries that `resync` doesn't fix create stale entries that `resync` doesn't fix
- **OC reboot breaks Proxmox VMs**: guest reboot corrupts TPM encryption - **OC reboot breaks Proxmox VMs**: guest reboot likely corrupts dqlite/Incus
database (open research item -- OS boots but Incus daemon doesn't start)
- **Token seed format quirk**: requires structured YAML with section keys, - **Token seed format quirk**: requires structured YAML with section keys,
not flat format not flat format
- **Expired tokens not auto-cleaned**: remain in list after expiry - **Expired tokens not auto-cleaned**: remain in list after expiry
@ -798,7 +801,7 @@ ip neigh show | grep -i "$MAC" | awk '{print $1}'
| OC CLI "not authorized" | Wrong certs in `~/.config/operations-center/` | Copy `client.crt` and `client.key` from `~/.config/incus/` | | OC CLI "not authorized" | Wrong certs in `~/.config/operations-center/` | Copy `client.crt` and `client.key` from `~/.config/incus/` |
| Cluster Terraform errors | `apply_defaults: true` pre-created resources | Non-fatal -- cluster forms despite "already in trust store" / "not in pending state" errors | | Cluster Terraform errors | `apply_defaults: true` pre-created resources | Non-fatal -- cluster forms despite "already in trust store" / "not in pending state" errors |
| `incus` TLS error after clustering | Cluster cert regenerated | `incus remote remove <r>` then `incus remote add <r> https://IP:8443 --accept-certificate` | | `incus` TLS error after clustering | Cluster cert regenerated | `incus remote remove <r>` then `incus remote add <r> https://IP:8443 --accept-certificate` |
| OC reboot kills IncusOS node | TPM PCR corruption from guest reboot | Destroy and redeploy the VM. Use Proxmox stop/start instead of OC reboot. | | OC reboot kills IncusOS node | Likely dqlite/Incus database corruption from unclean shutdown (open research item) | Destroy and redeploy the VM. Use Proxmox stop/start instead of OC reboot. |
| OC stale server entry after node replace | OC doesn't detect out-of-band cluster changes | `cluster resync` doesn't fix stale entries. Must manage cluster lifecycle through OC. | | OC stale server entry after node replace | OC doesn't detect out-of-band cluster changes | `cluster resync` doesn't fix stale entries. Must manage cluster lifecycle through OC. |
| OC shows old IP for replaced node | Re-registration blocked by existing entry | Remove stale entry (requires cluster dissociation first) | | OC shows old IP for replaced node | Re-registration blocked by existing entry | Remove stale entry (requires cluster dissociation first) |
| Token expired but still listed | Tokens don't auto-clean | Remove with `provisioning token remove <UUID>` | | Token expired but still listed | Tokens don't auto-clean | Remove with `provisioning token remove <UUID>` |
@ -847,12 +850,39 @@ ip neigh show | grep -i "$MAC" | awk '{print $1}'
- Manual: full control, no extra server, works with any Incus version - Manual: full control, no extra server, works with any Incus version
- OC hybrid: auto-registration, single-command clustering, centralized - OC hybrid: auto-registration, single-command clustering, centralized
inventory/updates, but requires OC server + OC-provisioned ISOs inventory/updates, but requires OC server + OC-provisioned ISOs
- OC reboot command is **dangerous** on Proxmox (breaks TPM-based encryption) - OC reboot command is **dangerous** on Proxmox (likely corrupts dqlite database)
- OC inventory requires manual resync (not real-time) - OC inventory requires manual resync (not real-time)
- Out-of-band cluster changes (via `incus` CLI) create stale OC state - Out-of-band cluster changes (via `incus` CLI) create stale OC state
--- ---
## Open Research Items
These are observed behaviors whose root cause is not yet confirmed:
- **OC reboot failure mechanism**: guest-level (ACPI) reboot on Proxmox VMs
causes the Incus daemon to never start. The OS boots (node is pingable,
TPM disk decryption succeeds) but port 8443 never opens. Likely cause:
dqlite database corruption from unclean shutdown. Proxmox stop/start does
not recover the node. Needs investigation with IncusOS debug logs.
- **apply_defaults conflict with OC Terraform**: when nodes have
`apply_defaults: true`, OC's post-clustering Terraform step fails with
"already in trust store" / "not in pending state" errors. The cluster forms
successfully despite these errors. Unclear whether skipping `apply_defaults`
would allow clean Terraform completion (untested).
- **Stale server entries after out-of-band changes**: when cluster members are
removed via `incus cluster remove` (not through OC), stale entries persist in
OC's server list. `cluster resync` doesn't fix them. The entry can't be
removed while OC considers it part of a cluster.
- **Inventory sync is manual**: OC inventory requires explicit `cluster resync`
to pick up changes made via the `incus` CLI. No webhook or event-driven sync.
This applies to instance creation, migration, evacuation, and deletion.
---
## References ## References
- [Operations Center GitHub](https://github.com/FuturFusion/operations-center) - [Operations Center GitHub](https://github.com/FuturFusion/operations-center)

263
notes/utm-support.md Normal file
View File

@ -0,0 +1,263 @@
# UTM Support Design Document
Design notes for a future `incusos-utm` script (or `--backend utm` flag on
`incusos-proxmox`) that would deploy IncusOS VMs on macOS using UTM as the
hypervisor.
**Status**: design only. Implementation requires macOS with UTM installed.
---
## Why UTM
UTM is a macOS-native virtualization app built on Apple's Hypervisor.framework
(for arm64 VMs) and QEMU (for x86_64 VMs). It provides a GUI and a CLI
(`utmctl`) for VM management.
For macOS users who want to run an IncusOS lab locally (instead of on a remote
Proxmox host), UTM is the most accessible option. It supports:
- UEFI boot (required by IncusOS)
- TPM 2.0 emulation (required for disk encryption)
- VirtIO devices (required by IncusOS)
- Nested virtualization (via Apple Hypervisor.framework)
- ISO attachment for boot media
---
## Architecture: UTM vs Proxmox
| Feature | Proxmox | UTM |
|---------|---------|-----|
| VM creation | API (`POST /qemu`) or SSH (`qm create`) | `utmctl` CLI or AppleScript |
| Disk management | API-managed, VMID-based naming | File-based (`.utm` bundles) |
| ISO attach | API (`ide2`, `ide3` config keys) | GUI or utmctl, possibly AppleScript |
| Boot monitoring | API (`blockstat.scsi0.wr_bytes` polling) | No equivalent -- use timeout + port polling |
| IP detection | ARP after MAC lookup from API | ARP after MAC from `utmctl` or VM config |
| UEFI/TPM | `bios=ovmf`, `tpmstate0` config keys | GUI settings per VM |
| Resource pool | Proxmox resource pools | No equivalent (single-user) |
| Remote access | SSH or REST API over network | Local only (no remote API) |
| Automation | Full API + SSH, headless | `utmctl` CLI, AppleScript, limited headless |
---
## UTM CLI: utmctl
UTM provides `utmctl` for basic VM management:
```bash
# List VMs
utmctl list
# Start/stop
utmctl start <vm-name>
utmctl stop <vm-name>
# Get VM info (includes IP if guest agent is available)
utmctl status <vm-name>
# Clone
utmctl clone <vm-name> --name <new-name>
# Delete
utmctl delete <vm-name>
# Attach USB device
utmctl usb connect <vm-name> <device>
```
### Limitations of utmctl
- **No VM creation**: VMs must be created via GUI or AppleScript
- **No disk management**: cannot attach/detach ISOs programmatically
- **No hardware config**: cannot set CPU count, memory, TPM, etc.
- **No boot order control**: must use GUI
These limitations mean a fully automated deployment (like `incusos-proxmox`)
would require AppleScript for VM creation and configuration, with `utmctl`
for start/stop/status operations.
---
## Proposed Implementation
### Option A: Separate script (`incusos-utm`)
A dedicated script that handles UTM-specific VM creation and management.
Reuses seed generation from `incusos-seed`.
```
incusos-utm [OPTIONS] CONFIG_FILE
--dry-run Preview actions
--status Check VM status
--cleanup Delete VMs
--lab-up Start stopped VMs
--lab-down Stop running VMs
```
### Option B: Backend flag on `incusos-proxmox`
Add `--backend utm` to the existing script. This requires abstracting the
Proxmox-specific functions behind a backend interface.
```
incusos-proxmox --backend utm CONFIG_FILE
```
**Recommendation**: Option A (separate script) is simpler and avoids
complicating the well-tested Proxmox workflow. The scripts can share
config format and seed generation.
---
## VM Creation via AppleScript
Since `utmctl` cannot create VMs, we'd use AppleScript:
```applescript
tell application "UTM"
-- Create a new VM
set newVM to make new virtual machine with properties {
name: "incus-lab-01",
backend: QEMU,
architecture: x86_64
}
-- Configure hardware
set memory of newVM to 4096
set cpu cores of newVM to 4
-- Add drives
-- (AppleScript API for UTM disk management TBD)
end tell
```
**Note**: UTM's AppleScript API may not expose all hardware settings.
Alternative: use UTM's `.utm` bundle format (plist + disk files) to
create VM bundles programmatically.
---
## Install Detection Without blockstat
Proxmox provides real-time disk I/O statistics via API, which
`incusos-proxmox` uses to detect when IncusOS installation completes.
UTM has no equivalent.
### Proposed alternative: timeout + port polling
```bash
# 1. Start VM with boot ISO + SEED_DATA
utmctl start "$vm_name"
# 2. Wait for expected install duration (2-5 minutes)
sleep 120
# 3. Poll port 8443 until it responds (with timeout)
local elapsed=0
while [[ $elapsed -lt 300 ]]; do
if nc -z "$vm_ip" 8443 2>/dev/null; then
echo "IncusOS is up"
break
fi
sleep 10
elapsed=$((elapsed + 10))
done
# 4. If force_reboot is in the seed, the VM reboots after install.
# On reboot, stop the VM, remove ISOs, restart from disk.
```
### IP Detection
UTM VMs on the default shared network get IPs from macOS's built-in DHCP.
Detection approaches:
1. **utmctl status**: may report IP if QEMU guest agent is installed
(IncusOS doesn't have guest agent)
2. **ARP lookup**: same as Proxmox -- get MAC from VM config, scan ARP table
3. **DHCP lease file**: check `/var/db/dhcpd_leases` on macOS
---
## SEED_DATA Delivery
Two options for attaching the SEED_DATA to a UTM VM:
1. **ISO on virtual CD-ROM**: generate ISO with `incusos-seed --format iso`,
attach as secondary CD-ROM in UTM
2. **External drive**: generate FAT image with `incusos-seed --format fat`,
attach as secondary disk
ISO is preferred (matches the Proxmox workflow).
---
## Config File Compatibility
The YAML config format should be compatible with `incusos-proxmox` configs
(minus the `proxmox:` section). UTM-specific settings could go in a `utm:`
section:
```yaml
# utm: section (optional, UTM-specific overrides)
utm:
backend: qemu # qemu or apple (default: qemu)
network: shared # shared, bridged, host-only
display: true # show VM window (default: true for UTM)
defaults:
cores: 4
memory: 4096
disk: 50
vms:
- name: incus-lab-01
app: incus
apply_defaults: true
```
---
## Limitations and Trade-offs
| Aspect | Proxmox | UTM |
|--------|---------|-----|
| Automation level | Full (API/SSH) | Partial (utmctl + AppleScript) |
| Headless operation | Yes | Limited (UTM needs to be running) |
| Remote deployment | Yes (network API) | No (local only) |
| Multi-node scaling | Tested up to 4 VMs | Limited by host RAM/CPU |
| Install monitoring | Real-time (blockstat) | Timeout-based |
| Resource isolation | Proxmox pools | None |
| Nested virt | Intel VT-x passthrough | Apple HV.framework (arm64) |
### When to use which
- **Proxmox**: production labs, multi-node clusters, automated CI, remote
management, teams
- **UTM**: local development, single-developer labs, macOS-only environments,
quick experiments
---
## Implementation Roadmap
1. **Research phase** (now): document UTM capabilities, `utmctl` API surface,
AppleScript integration
2. **Seed generation**: already works (`incusos-seed` is cross-platform
after Phase 3 macOS compatibility)
3. **VM creation**: implement AppleScript or `.utm` bundle generation
4. **Install flow**: implement timeout-based install detection
5. **Status/lifecycle**: implement `--status`, `--lab-up`, `--lab-down`
using `utmctl`
6. **Testing**: test on macOS with Apple Silicon and Intel Macs
---
## References
- [UTM documentation](https://docs.getutm.app/)
- [utmctl CLI](https://docs.getutm.app/advanced/remote-control/)
- [UTM scripting](https://docs.getutm.app/scripting/scripting/)
- [Apple Hypervisor.framework](https://developer.apple.com/documentation/hypervisor)