22 KiB

Raw Blame History

Testing Guide: incusos-proxmox

A step-by-step guide for deploying IncusOS VMs to Proxmox VE using incusos-proxmox.

Implementation Status

The three scripts (incusos-proxmox, incusos-seed, incusos-iso) implement the full lifecycle:

Config parsing -- YAML with PyYAML or built-in fallback parser
ISO download -- fetches latest IncusOS from the CDN by channel (stable/testing)
Per-VM seed generation -- calls incusos-seed --format iso for each VM
Upload to Proxmox -- SCP (SSH method) or multipart POST (API method)
VM creation -- all IncusOS-required settings enforced (UEFI, TPM, VirtIO-SCSI, etc.)
Automated install monitoring -- polls blockstat.scsi0.wr_bytes via API to detect when disk writes stop (3 stable polls = 15s idle), then stops VM, detaches install media (ide2 + ide3), and boots from disk
Cleanup -- safe teardown with managed-VM marker checks

Cluster Formation (Manual Step)

The cluster and cluster_role fields are accepted in config for documentation purposes, but automatic cluster join is not yet implemented. After deployment, join nodes to the cluster manually:

# On the init node:
incus cluster add <node-name>
# Copy the join token, then on the joining node's preseed or CLI

Authentication

SSH (Default)

Uses key-based SSH as root@<host>. Requires no setup beyond ssh-copy-id.

API Token (Recommended for Production)

Avoids root SSH entirely. Uses a dedicated Proxmox user with minimal privileges.

Minimum Required Privileges

The role below is already the minimum viable set -- every privilege is required by at least one operation in the script:

Privilege	Used For
`VM.Allocate`	Create and destroy VMs
`VM.Config.Disk`	scsi0, efidisk0, tpmstate0
`VM.Config.CPU`	Set cores, cpu=host
`VM.Config.Memory`	Set memory, balloon=0
`VM.Config.Network`	Attach VM to bridge
`VM.Config.CDROM`	Attach ISO (ide2) and seed (ide3)
`VM.Config.Options`	Set boot order, description, agent flag
`VM.Config.HWType`	Set bios=ovmf, machine=q35, scsihw
`VM.PowerMgmt`	Start, stop, and reboot VMs
`VM.Audit`	Query VM status and config
`Datastore.AllocateSpace`	Create disks in storage pool
`Datastore.AllocateTemplate`	Upload ISOs to storage
`Datastore.Audit`	List and check storage pools
`SDN.Use`	Use network bridges

None of these can be removed without breaking functionality.

Setup Commands (Run on Proxmox)

pveum role add IncusOSDeploy --privs "VM.Allocate VM.Config.Disk \
    VM.Config.CPU VM.Config.Memory VM.Config.Network VM.Config.CDROM \
    VM.Config.Options VM.Config.HWType VM.PowerMgmt VM.Audit \
    Datastore.AllocateSpace Datastore.AllocateTemplate \
    Datastore.Audit SDN.Use"

pveum user add automation@pve
pveum aclmod / -user automation@pve -role IncusOSDeploy
pveum user token add automation@pve deploy --privsep 0
# Save the displayed secret!

Optional: Scoped ACLs

For tighter access control, scope the ACL to specific paths instead of /:

pveum aclmod /storage/local -user automation@pve -role IncusOSDeploy
pveum aclmod /storage/local-lvm -user automation@pve -role IncusOSDeploy
pveum aclmod /vms -user automation@pve -role IncusOSDeploy

For a home lab, / (root) is simpler and perfectly acceptable.

Guided Test Scenario

Prerequisites

On your workstation (where you run the script):

# Verify tools
bash --version           # Need 4+
python3 --version        # For YAML parsing
which jq curl ssh scp    # All needed for SSH method
which genisoimage        # Needed for seed ISO images (used by incusos-proxmox)

# Verify the Incus client has generated a keypair
ls ~/.config/incus/client.crt ~/.config/incus/client.key
# If missing, run:  incus remote list   (triggers auto-generation)

On your Proxmox host: nothing to install -- just ensure SSH key auth works.

Step 1: Verify SSH Connectivity

ssh root@<YOUR_PROXMOX_IP> pvesh get /version

You should see JSON with the Proxmox VE version. If this fails, fix SSH key auth first (ssh-copy-id root@<host>).

Step 2: Set Up Proxmox Connection

cp incusos/examples/proxmox.yaml.example incusos/proxmox.yaml

Edit incusos/proxmox.yaml with your Proxmox host settings. This is auto-discovered by the scripts and avoids repeating credentials in each lab config.

Step 3: Create a Test Config

cp incusos/examples/proxmox-minimal.yaml my-test.yaml

Edit my-test.yaml -- change only the host IP (or rely on proxmox.yaml):

proxmox:
  host: <YOUR_PROXMOX_IP>       # <-- change this

defaults:
  start_vmid: 900                # High range to avoid collisions

vms:
  - name: incus-test
    app: incus
    apply_defaults: true
    cores: 4
    memory: 8192
    disk: 64

Using VMID 900+ avoids clashing with existing VMs.

Step 4: Dry Run

./incusos/incusos-proxmox --dry-run my-test.yaml

Verify in the output:

Config parsed correctly (host, node, storage shown)
Client certificate detected (~/.config/incus/client.crt)
Storage pools local-lvm and local found (adjust for your setup)
Network bridge vmbr0 found
VMID 900 allocated for incus-test
ISO version resolved from CDN (e.g., IncusOS_25.03.iso)
[dry-run] prefix on all actions -- nothing actually executed
No errors

Step 5: Deploy (Single VM)

./incusos/incusos-proxmox my-test.yaml

Watch the phases:

Validate -- connects to Proxmox, checks storage/bridge/tools
Confirm -- type y to proceed
Prepare -- downloads IncusOS ISO from CDN (~500 MB), generates seed
Upload -- SCPs ISO + seed to Proxmox
Create -- runs qm create 900 ... with all IncusOS settings
Install -- starts VM, monitors disk I/O via blockstat, detaches media, boots from disk (~2-3 min)

Simultaneously on the Proxmox web UI:

Open https://<YOUR_PROXMOX_IP>:8006
VM 900 (incus-test) should appear
Open its Console tab
Watch the IncusOS installer boot from ISO, read the seed, install to disk, and reboot
After reboot you should see the IncusOS login prompt

Step 6: Check Deployment Status

After the deploy completes, verify with the built-in status command:

./incusos/incusos-proxmox --status my-test.yaml

Verify in the output:

VM incus-test exists and is managed
Status is running, install state is installed
IP address detected
Port 8443 is open
Incus remote status shown (if configured)

Step 7: Verify the Deployed VM

Once the script reports success with an IP address:

# The cert was injected via seed, so no auth prompt appears
incus remote add incus-test <REPORTED_IP> --accept-certificate

# Test connectivity
incus remote list
incus info incus-test:

# If apply_defaults was true, there should be a ZFS pool and bridge
incus storage list incus-test:
incus network list incus-test:

Note: IncusOS is immutable and has no QEMU guest agent. IP detection uses ARP-based lookup (MAC from Proxmox VM config → ARP table). If the IP is wrong, check the Proxmox console — stale ARP entries can be misleading.

Step 8: Test Re-run Reconcile

Re-run the deploy to verify the reconcile menu:

./incusos/incusos-proxmox my-test.yaml

Verify: The script detects the existing VM and shows the interactive menu. Choose option 1 to run status checks without modifying anything.

Also test with --yes to verify it defaults to safe behavior:

./incusos/incusos-proxmox --yes my-test.yaml

Verify: Runs post-deployment checks, does NOT destroy.

Step 9: Clean Up the Test VM

./incusos/incusos-proxmox --cleanup my-test.yaml

Confirms before destroying
Checks the management marker ([incusos-lab:managed] in the VM description)
Stops and destroys with --purge

Remove the remote too:

incus remote remove incus-test

Step 10: (Optional) Test API Method

After single-VM success with SSH, switch to API tokens.

On Proxmox (one-time setup via SSH or web shell):

pveum role add IncusOSDeploy --privs "VM.Allocate VM.Config.Disk \
    VM.Config.CPU VM.Config.Memory VM.Config.Network VM.Config.CDROM \
    VM.Config.Options VM.Config.HWType VM.PowerMgmt VM.Audit \
    Datastore.AllocateSpace Datastore.AllocateTemplate \
    Datastore.Audit SDN.Use"
pveum user add automation@pve
pveum aclmod / -user automation@pve -role IncusOSDeploy
pveum user token add automation@pve deploy --privsep 0
# ^^^ Save the displayed secret!

On your workstation:

export PROXMOX_TOKEN_SECRET="<the-secret-from-above>"

Update my-test.yaml:

proxmox:
  host: <YOUR_PROXMOX_IP>
  method: api
  api_token_id: automation@pve!deploy

Repeat steps 4-7 using the API method.

Step 11: (Optional) Test with Resource Pool

For API method with pool isolation:

On Proxmox:

pveum pool add IncusLab --comment "IncusOS Lab VMs"
pveum pool modify IncusLab --storage local-lvm,local
pveum aclmod /pool/IncusLab -user automation@pve -role IncusOSDeploy

Add to your config:

proxmox:
  pool: IncusLab

Deploy and verify:

./incusos/incusos-proxmox --dry-run my-test.yaml   # Should show pool in plan
./incusos/incusos-proxmox my-test.yaml              # VM created in pool
./incusos/incusos-proxmox --status my-test.yaml     # Status scoped to pool

Step 12: (Optional) Full Lab Deployment

Once single-VM tests pass, deploy the full 4-VM lab:

cp incusos/examples/proxmox-lab.yaml my-lab.yaml
# Edit: change host IP, adjust start_vmid if needed
./incusos/incusos-proxmox --dry-run my-lab.yaml
./incusos/incusos-proxmox my-lab.yaml

This deploys 1 Operations Center + 3 Incus nodes. After deployment:

Access Operations Center via browser at its reported IP
On the init node, run incus cluster add <node-name> to get join tokens
Join the other two nodes using the tokens

Lab Validation with lab-test

The lab-test script automates the end-to-end validation of deployed IncusOS VMs: single-node workloads, cluster formation, migration, and evacuation.

Quick Start: Cluster Testing

# 1. Check environment
./incusos/incusos-proxmox --doctor

# 2. Deploy 3 nodes
./incusos/incusos-proxmox --yes examples/lab-cluster.yaml

# 3. Add incus remotes for each node (use IPs from --status)
./incusos/incusos-proxmox --status examples/lab-cluster.yaml
incus remote add incus-lab-01 <IP1> --accept-certificate
incus remote add incus-lab-02 <IP2> --accept-certificate
incus remote add incus-lab-03 <IP3> --accept-certificate

# 4. Run full lab validation
./incusos/lab-test --yes examples/lab-cluster.yaml

Phase-by-Phase Testing

# Single-node workloads only (container + VM launch/exec)
./incusos/lab-test --phase single --skip-deploy examples/lab-cluster.yaml

# Cluster formation only
./incusos/lab-test --phase cluster --skip-deploy examples/lab-cluster.yaml

# Cluster workloads (scheduler placement + targeted placement)
./incusos/lab-test --phase workload --skip-deploy examples/lab-cluster.yaml

# Migration and evacuation tests
./incusos/lab-test --phase migrate --skip-deploy examples/lab-cluster.yaml

Manual Cluster Formation (Alternative)

If you prefer to form the cluster manually instead of using lab-test, follow the steps below. See also notes/clustering-guide.md for the full reference guide with explanations of what happens under the hood.

Prerequisites

# Check client version (need 6.20+ for remote cluster join)
incus version

# Verify all three remotes are working
incus info incus-lab-01: | head -5
incus info incus-lab-02: | head -5
incus info incus-lab-03: | head -5

Step 1: Set specific IPs on all nodes

IncusOS defaults to core.https_address: :8443 (wildcard). Clustering requires a specific routable IP on each node.

# Discover each node's routable IP
for node in incus-lab-01 incus-lab-02 incus-lab-03; do
  echo "=== $node ==="
  incus query "$node:/1.0" | python3 -c "import sys,json; d=json.load(sys.stdin); \
    [print(a) for a in d['environment']['addresses'] \
     if not a.startswith('10.') and not a.startswith('fd42:') and not a.startswith('[')]"
done

# Set the specific IP on each node (replace with actual IPs)
incus config set incus-lab-01: core.https_address <IP1>:8443
incus config set incus-lab-02: core.https_address <IP2>:8443
incus config set incus-lab-03: core.https_address <IP3>:8443

# Verify connectivity still works after each change
incus info incus-lab-01: | head -3
incus info incus-lab-02: | head -3
incus info incus-lab-03: | head -3

Step 2: Enable clustering on the init node

incus cluster enable incus-lab-01:incus-lab-01

Important: This regenerates the server's TLS certificate. Your remote will break with a certificate error. Fix it:

incus remote switch local          # if incus-lab-01 is the current default
incus remote remove incus-lab-01
incus remote add incus-lab-01 https://<IP1>:8443 --accept-certificate

Verify:

incus cluster list incus-lab-01:
# Should show one member (incus-lab-01), status ONLINE, role database-leader

Step 3: Prepare joining nodes

Nodes with apply_defaults: true have a pre-existing local storage pool and incusbr0 network. These conflict with the cluster join process (the join wizard tries to set member-specific ZFS config keys at cluster level). Delete them before joining:

# For each joining node (incus-lab-02, incus-lab-03):
NODE=incus-lab-02   # then repeat with incus-lab-03

# Remove server-level references to the pool
incus config unset "$NODE": storage.backups_volume
incus config unset "$NODE": storage.images_volume

# Delete the backup and image volumes
incus storage volume delete "$NODE":local backups
incus storage volume delete "$NODE":local images

# Clear default profile references (otherwise pool is "in use")
incus profile device remove "$NODE":default root
incus profile device remove "$NODE":default eth0

# Delete pool and network
incus storage delete "$NODE":local
incus network delete "$NODE":incusbr0

Step 4: Join nodes to the cluster

Interactive (one node at a time):

# Generate join token
incus cluster add incus-lab-01:incus-lab-02

# Join (answer the interactive prompts)
incus cluster join incus-lab-01: incus-lab-02:
# Prompt 1 - IP address: press Enter (accept default)
# Prompt 2 - Member name: press Enter (accept default)
# Prompt 3 - "All existing data is lost": type "yes"
# Prompt 4 - source for storage pool "local": type "local/incus"
# Prompt 5 - zfs.pool_name for storage pool "local": type "local/incus"

Automated (non-interactive):

# Generate token + join in one go
incus cluster add incus-lab-01:incus-lab-02
printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join incus-lab-01: incus-lab-02:

After each join, fix the remote (same certificate issue as the init node):

incus remote remove incus-lab-02
incus remote add incus-lab-02 https://<IP2>:8443 --accept-certificate

Repeat for incus-lab-03.

Step 5: Verify the cluster

# All 3 members should be ONLINE
incus cluster list incus-lab-01:

# Storage pool should show on all members
incus storage show incus-lab-01:local
incus storage show incus-lab-01:local --target incus-lab-01
incus storage show incus-lab-01:local --target incus-lab-02
incus storage show incus-lab-01:local --target incus-lab-03

# All remotes should work
incus info incus-lab-01: | head -3
incus info incus-lab-02: | head -3
incus info incus-lab-03: | head -3

Key points:

apply_defaults: true on all nodes ensures matching storage pool and network bridge names (required for migration) -- but the pools must be deleted on joining nodes before the join
After joining, manage the cluster through the init node's remote
The ZFS pool source is local/incus on IncusOS (auto-created by apply_defaults)
Each join generates a new token (tokens are single-use and time-limited)
VM migration requires matching storage pool names across nodes

Migration Testing (After Cluster Formation)

Once the cluster is formed, validate container and VM migration:

Container migration (stop/move/start)

# Launch on node 2
incus launch images:debian/12 incus-lab-01:test-container --target incus-lab-02
incus exec incus-lab-01:test-container -- bash -c 'echo "hello" > /root/test.txt'

# Migrate to node 3
incus stop incus-lab-01:test-container
incus move incus-lab-01:test-container --target incus-lab-03
incus start incus-lab-01:test-container

# Verify: data survives, processes don't
incus exec incus-lab-01:test-container -- cat /root/test.txt   # "hello"
incus list incus-lab-01: --columns nLs                         # node 3

VM live migration

# Launch and configure for migration
incus launch images:debian/12 incus-lab-01:test-vm --vm --target incus-lab-02
# Wait for VM agent...
incus stop incus-lab-01:test-vm
incus config set incus-lab-01:test-vm migration.stateful=true
incus config set incus-lab-01:test-vm limits.cpu=0-1
incus config device add incus-lab-01:test-vm root disk path=/ pool=local size.state=2GiB
incus start incus-lab-01:test-vm

# Set up heartbeat for state verification
incus exec incus-lab-01:test-vm -- mkdir -p /tmp/migration-test
incus exec incus-lab-01:test-vm -- bash -c \
  'cat > /tmp/migration-test/heartbeat.sh << "EOF"
#!/bin/bash
i=0; while true; do echo $i > /tmp/migration-test/heartbeat; i=$((i+1)); sleep 1; done
EOF
chmod +x /tmp/migration-test/heartbeat.sh'
incus exec incus-lab-01:test-vm -- \
  systemd-run --unit=migration-heartbeat /tmp/migration-test/heartbeat.sh

# Live-migrate (state preserved)
sleep 3
incus exec incus-lab-01:test-vm -- cat /tmp/migration-test/heartbeat   # note value
incus move incus-lab-01:test-vm --target incus-lab-03
sleep 2   # agent needs ~1-2s to reconnect
incus exec incus-lab-01:test-vm -- cat /tmp/migration-test/heartbeat   # should be higher
incus exec incus-lab-01:test-vm -- systemctl is-active migration-heartbeat  # "active"

Cluster evacuation

# Move both workloads to node 2, then evacuate
incus cluster evacuate incus-lab-01:incus-lab-02 --force
incus cluster list incus-lab-01:    # node 2 = EVACUATED
incus list incus-lab-01: --columns nLs  # workloads on other nodes

# Restore
incus cluster restore incus-lab-01:incus-lab-02 --force
incus cluster list incus-lab-01:    # node 2 = ONLINE
incus list incus-lab-01: --columns nLs  # workloads back on node 2

See notes/clustering-guide.md for the full automated migration test script and detailed test results.

Cleanup

# Remove test instances only (keeps VMs and cluster)
./incusos/lab-test --cleanup examples/lab-cluster.yaml

# Destroy VMs only
./incusos/incusos-proxmox --cleanup examples/lab-cluster.yaml

# Full cleanup: VMs + deployment ISO + seeds + remotes + cache
./incusos/incusos-proxmox --cleanup --deep --yes examples/lab-cluster.yaml

# Pool-wide: destroy ALL managed VMs (no config file needed)
./incusos/incusos-proxmox --cleanup-all --yes
./incusos/incusos-proxmox --cleanup-all --deep --yes   # nuke everything

Operations Center Lab

Deploy

./incusos/incusos-proxmox --yes examples/lab-oc.yaml
./incusos/incusos-proxmox --status examples/lab-oc.yaml

Access the Web UI

# Export client certificate for browser
openssl pkcs12 -export \
    -inkey ~/.config/incus/client.key \
    -in ~/.config/incus/client.crt \
    -out client.pfx

# Import client.pfx into Firefox, navigate to https://<OC_IP>:8443

OC CLI Setup

# Copy incus client certs for OC CLI
mkdir -p ~/.config/operations-center
cp ~/.config/incus/client.crt ~/.config/operations-center/
cp ~/.config/incus/client.key ~/.config/operations-center/

# Add OC remote
operations-center remote add lab-oc https://<OC_IP>:8443
operations-center remote switch lab-oc

# Check for provisioning updates
operations-center provisioning update list

Provisioning Workflow

# Create a provisioning token
operations-center provisioning token add --description "lab nodes" --uses 10

# Get provisioned ISO
operations-center provisioning token get-image <token> ~/Downloads/IncusOS-provisioned.iso

# Redeploy nodes with provisioned ISO, or check if manual adoption works

Note: Operations Center is under active development (v0.2.2+). Commands and APIs may change between versions. The --doctor command reports whether the OC CLI is installed.

Troubleshooting

Symptom	Likely Cause	Fix
SSH connection refused	Key auth not set up	`ssh-copy-id root@<host>`
Storage pool not found	Wrong pool name in config	Check `pvesm status` on Proxmox
ISO download fails	CDN unreachable	Download manually, use `--iso path/to/file.iso`
VM creation fails	VMID collision	Use higher `start_vmid` or check `qm list`
Install times out (10 min)	Slow I/O or boot issue	Check VM console in Proxmox web UI
No IP reported	No guest agent (expected)	Check Proxmox console; script uses ARP-based detection
Wrong IP reported	Stale ARP entries	Script flushes ARP, but verify via Proxmox console
"install media detected" on boot	ide2/ide3 still attached	Script auto-detaches; if manual, delete ide2+ide3 then start
"no target device matched"	Explicit `disk-target` in seed	Remove `disk-target`; let IncusOS auto-detect
"field server not found" YAML error	Wrong preseed structure	Use `preseed.certificates[]` not `preseed.server.certificates[]`
Cert not auto-detected	No Incus keypair	Run `incus remote list` to generate one
`genisoimage` not found	genisoimage not installed	Install `genisoimage` package
Pool not found	Resource pool not created	Run `pveum pool add <name>` on Proxmox
Re-run destroys VMs	Used option 3 in reconcile	Use option 1 (checks) or option 2 (continue) instead
`--status` shows "not found"	VM was cleaned up	Deploy first, then check status
VM live migration fails: "Missing section footer for ICH9LPC"	`limits.cpu` not set or set as integer	Use range syntax: `limits.cpu=0-1` (not `2`); see `notes/clustering-guide.md`
`migration.stateful` cannot be set while running	Incus requires VM to be stopped	Stop VM first, set config, then start
`certificate is valid for 127.0.0.1, ::1, not <IP>`	Cluster cert regenerated on enable/join	Remove and re-add remote with `--accept-certificate`
`Config key "zfs.pool_name" is cluster member specific`	Storage pool exists on joining node	Delete pool/network before joining; see clustering guide

22 KiB Raw Blame History

Testing Guide: incusos-proxmox

Implementation Status

Cluster Formation (Manual Step)

Authentication

SSH (Default)

API Token (Recommended for Production)

Minimum Required Privileges

Setup Commands (Run on Proxmox)

Optional: Scoped ACLs

Guided Test Scenario

Prerequisites

Step 1: Verify SSH Connectivity

Step 2: Set Up Proxmox Connection

Step 3: Create a Test Config

Step 4: Dry Run

Step 5: Deploy (Single VM)

Step 6: Check Deployment Status

Step 7: Verify the Deployed VM

Step 8: Test Re-run Reconcile

Step 9: Clean Up the Test VM

Step 10: (Optional) Test API Method

Step 11: (Optional) Test with Resource Pool

Step 12: (Optional) Full Lab Deployment

Lab Validation with lab-test

Quick Start: Cluster Testing

Phase-by-Phase Testing

Manual Cluster Formation (Alternative)

Prerequisites

Step 1: Set specific IPs on all nodes

Step 2: Enable clustering on the init node

Step 3: Prepare joining nodes

Step 4: Join nodes to the cluster

Step 5: Verify the cluster

Migration Testing (After Cluster Formation)

Container migration (stop/move/start)

VM live migration

Cluster evacuation

Cleanup

Operations Center Lab

Deploy

Access the Web UI

OC CLI Setup

Provisioning Workflow

Troubleshooting

22 KiB

Raw Blame History