maarten
/
incus-contrib
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
incus-contrib/incusos/TESTING.md

15 KiB
Raw Blame History

Testing Guide: incusos-proxmox

A step-by-step guide for deploying IncusOS VMs to Proxmox VE using incusos-proxmox.

Implementation Status

The three scripts (incusos-proxmox, incusos-seed, incusos-iso) implement the full lifecycle:

  • Config parsing -- YAML with PyYAML or built-in fallback parser
  • ISO download -- fetches latest IncusOS from the CDN by channel (stable/testing)
  • Per-VM seed generation -- calls incusos-seed --format iso for each VM
  • Upload to Proxmox -- SCP (SSH method) or multipart POST (API method)
  • VM creation -- all IncusOS-required settings enforced (UEFI, TPM, VirtIO-SCSI, etc.)
  • Automated install monitoring -- polls blockstat.scsi0.wr_bytes via API to detect when disk writes stop (3 stable polls = 15s idle), then stops VM, detaches install media (ide2 + ide3), and boots from disk
  • Cleanup -- safe teardown with managed-VM marker checks

Cluster Formation (Manual Step)

The cluster and cluster_role fields are accepted in config for documentation purposes, but automatic cluster join is not yet implemented. After deployment, join nodes to the cluster manually:

# On the init node:
incus cluster add <node-name>
# Copy the join token, then on the joining node's preseed or CLI

Authentication

SSH (Default)

Uses key-based SSH as root@<host>. Requires no setup beyond ssh-copy-id.

API Token (Recommended for Production)

Avoids root SSH entirely. Uses a dedicated Proxmox user with minimal privileges.

Minimum Required Privileges

The role below is already the minimum viable set -- every privilege is required by at least one operation in the script:

Privilege Used For
VM.Allocate Create and destroy VMs
VM.Config.Disk scsi0, efidisk0, tpmstate0
VM.Config.CPU Set cores, cpu=host
VM.Config.Memory Set memory, balloon=0
VM.Config.Network Attach VM to bridge
VM.Config.CDROM Attach ISO (ide2) and seed (ide3)
VM.Config.Options Set boot order, description, agent flag
VM.Config.HWType Set bios=ovmf, machine=q35, scsihw
VM.PowerMgmt Start, stop, and reboot VMs
VM.Audit Query VM status and config
Datastore.AllocateSpace Create disks in storage pool
Datastore.AllocateTemplate Upload ISOs to storage
Datastore.Audit List and check storage pools
SDN.Use Use network bridges

None of these can be removed without breaking functionality.

Setup Commands (Run on Proxmox)

pveum role add IncusOSDeploy --privs "VM.Allocate VM.Config.Disk \
    VM.Config.CPU VM.Config.Memory VM.Config.Network VM.Config.CDROM \
    VM.Config.Options VM.Config.HWType VM.PowerMgmt VM.Audit \
    Datastore.AllocateSpace Datastore.AllocateTemplate \
    Datastore.Audit SDN.Use"

pveum user add automation@pve
pveum aclmod / -user automation@pve -role IncusOSDeploy
pveum user token add automation@pve deploy --privsep 0
# Save the displayed secret!

Optional: Scoped ACLs

For tighter access control, scope the ACL to specific paths instead of /:

pveum aclmod /storage/local -user automation@pve -role IncusOSDeploy
pveum aclmod /storage/local-lvm -user automation@pve -role IncusOSDeploy
pveum aclmod /vms -user automation@pve -role IncusOSDeploy

For a home lab, / (root) is simpler and perfectly acceptable.

Guided Test Scenario

Prerequisites

On your workstation (where you run the script):

# Verify tools
bash --version           # Need 4+
python3 --version        # For YAML parsing
which jq curl ssh scp    # All needed for SSH method
which genisoimage        # Needed for seed ISO images (used by incusos-proxmox)

# Verify the Incus client has generated a keypair
ls ~/.config/incus/client.crt ~/.config/incus/client.key
# If missing, run:  incus remote list   (triggers auto-generation)

On your Proxmox host: nothing to install -- just ensure SSH key auth works.

Step 1: Verify SSH Connectivity

ssh root@<YOUR_PROXMOX_IP> pvesh get /version

You should see JSON with the Proxmox VE version. If this fails, fix SSH key auth first (ssh-copy-id root@<host>).

Step 2: Create a Test Config

cp incusos/examples/proxmox-minimal.yaml my-test.yaml

Edit my-test.yaml -- change only the host IP:

proxmox:
  host: <YOUR_PROXMOX_IP>       # <-- change this

defaults:
  start_vmid: 900                # High range to avoid collisions

vms:
  - name: incus-test
    app: incus
    apply_defaults: true
    cores: 4
    memory: 8192
    disk: 64

Using VMID 900+ avoids clashing with existing VMs.

Step 3: Dry Run

./incusos/incusos-proxmox --dry-run my-test.yaml

Verify in the output:

  • Config parsed correctly (host, node, storage shown)
  • Client certificate detected (~/.config/incus/client.crt)
  • Storage pools local-lvm and local found (adjust for your setup)
  • Network bridge vmbr0 found
  • VMID 900 allocated for incus-test
  • ISO version resolved from CDN (e.g., IncusOS_25.03.iso)
  • [dry-run] prefix on all actions -- nothing actually executed
  • No errors

Step 4: Deploy (Single VM)

./incusos/incusos-proxmox my-test.yaml

Watch the phases:

  1. Validate -- connects to Proxmox, checks storage/bridge/tools
  2. Confirm -- type y to proceed
  3. Prepare -- downloads IncusOS ISO from CDN (~500 MB), generates seed
  4. Upload -- SCPs ISO + seed to Proxmox
  5. Create -- runs qm create 900 ... with all IncusOS settings
  6. Install -- starts VM, monitors disk I/O via blockstat, detaches media, boots from disk (~2-3 min)

Simultaneously on the Proxmox web UI:

  • Open https://<YOUR_PROXMOX_IP>:8006
  • VM 900 (incus-test) should appear
  • Open its Console tab
  • Watch the IncusOS installer boot from ISO, read the seed, install to disk, and reboot
  • After reboot you should see the IncusOS login prompt

Step 5: Check Deployment Status

After the deploy completes, verify with the built-in status command:

./incusos/incusos-proxmox --status my-test.yaml

Verify in the output:

  • VM incus-test exists and is managed
  • Status is running, install state is installed
  • IP address detected
  • Port 8443 is open
  • Incus remote status shown (if configured)

Step 6: Verify the Deployed VM

Once the script reports success with an IP address:

# The cert was injected via seed, so no auth prompt appears
incus remote add incus-test <REPORTED_IP> --accept-certificate

# Test connectivity
incus remote list
incus info incus-test:

# If apply_defaults was true, there should be a ZFS pool and bridge
incus storage list incus-test:
incus network list incus-test:

Note: IncusOS is immutable and has no QEMU guest agent. IP detection uses ARP-based lookup (MAC from Proxmox VM config → ARP table). If the IP is wrong, check the Proxmox console — stale ARP entries can be misleading.

Step 7: Test Re-run Reconcile

Re-run the deploy to verify the reconcile menu:

./incusos/incusos-proxmox my-test.yaml

Verify: The script detects the existing VM and shows the interactive menu. Choose option 1 to run status checks without modifying anything.

Also test with --yes to verify it defaults to safe behavior:

./incusos/incusos-proxmox --yes my-test.yaml

Verify: Runs post-deployment checks, does NOT destroy.

Step 8: Clean Up the Test VM

./incusos/incusos-proxmox --cleanup my-test.yaml
  • Confirms before destroying
  • Checks the management marker ([incusos-lab:managed] in the VM description)
  • Stops and destroys with --purge

Remove the remote too:

incus remote remove incus-test

Step 9: (Optional) Test API Method

After single-VM success with SSH, switch to API tokens.

On Proxmox (one-time setup via SSH or web shell):

pveum role add IncusOSDeploy --privs "VM.Allocate VM.Config.Disk \
    VM.Config.CPU VM.Config.Memory VM.Config.Network VM.Config.CDROM \
    VM.Config.Options VM.Config.HWType VM.PowerMgmt VM.Audit \
    Datastore.AllocateSpace Datastore.AllocateTemplate \
    Datastore.Audit SDN.Use"
pveum user add automation@pve
pveum aclmod / -user automation@pve -role IncusOSDeploy
pveum user token add automation@pve deploy --privsep 0
# ^^^ Save the displayed secret!

On your workstation:

export PROXMOX_TOKEN_SECRET="<the-secret-from-above>"

Update my-test.yaml:

proxmox:
  host: <YOUR_PROXMOX_IP>
  method: api
  api_token_id: automation@pve!deploy

Repeat steps 3-6 using the API method.

Step 10: (Optional) Test with Resource Pool

For API method with pool isolation:

On Proxmox:

pveum pool add IncusLab --comment "IncusOS Lab VMs"
pveum pool modify IncusLab --storage local-lvm,local
pveum aclmod /pool/IncusLab -user automation@pve -role IncusOSDeploy

Add to your config:

proxmox:
  pool: IncusLab

Deploy and verify:

./incusos/incusos-proxmox --dry-run my-test.yaml   # Should show pool in plan
./incusos/incusos-proxmox my-test.yaml              # VM created in pool
./incusos/incusos-proxmox --status my-test.yaml     # Status scoped to pool

Step 11: (Optional) Full Lab Deployment

Once single-VM tests pass, deploy the full 4-VM lab:

cp incusos/examples/proxmox-lab.yaml my-lab.yaml
# Edit: change host IP, adjust start_vmid if needed
./incusos/incusos-proxmox --dry-run my-lab.yaml
./incusos/incusos-proxmox my-lab.yaml

This deploys 1 Operations Center + 3 Incus nodes. After deployment:

  1. Access Operations Center via browser at its reported IP
  2. On the init node, run incus cluster add <node-name> to get join tokens
  3. Join the other two nodes using the tokens

Lab Validation with lab-test

The lab-test script automates the end-to-end validation of deployed IncusOS VMs: single-node workloads, cluster formation, migration, and evacuation.

Quick Start: Cluster Testing

# 1. Check environment
./incusos/incusos-proxmox --doctor

# 2. Deploy 3 nodes
./incusos/incusos-proxmox --yes examples/lab-cluster.yaml

# 3. Add incus remotes for each node (use IPs from --status)
./incusos/incusos-proxmox --status examples/lab-cluster.yaml
incus remote add incus-lab-01 <IP1> --accept-certificate
incus remote add incus-lab-02 <IP2> --accept-certificate
incus remote add incus-lab-03 <IP3> --accept-certificate

# 4. Run full lab validation
./incusos/lab-test --yes examples/lab-cluster.yaml

Phase-by-Phase Testing

# Single-node workloads only (container + VM launch/exec)
./incusos/lab-test --phase single --skip-deploy examples/lab-cluster.yaml

# Cluster formation only
./incusos/lab-test --phase cluster --skip-deploy examples/lab-cluster.yaml

# Cluster workloads (scheduler placement + targeted placement)
./incusos/lab-test --phase workload --skip-deploy examples/lab-cluster.yaml

# Migration and evacuation tests
./incusos/lab-test --phase migrate --skip-deploy examples/lab-cluster.yaml

Manual Cluster Formation (Alternative)

If you prefer to form the cluster manually instead of using lab-test:

# Step 1: Enable clustering on first node
incus cluster enable incus-lab-01: incus-lab-01

# Step 2: Generate join token for second node
incus cluster add incus-lab-01: incus-lab-02

# Step 3: Join second node
incus cluster join incus-lab-01: incus-lab-02:

# Step 4: Repeat for third node
incus cluster add incus-lab-01: incus-lab-03
incus cluster join incus-lab-01: incus-lab-03:

# Step 5: Verify
incus cluster list incus-lab-01:

Key points:

  • apply_defaults: true on all nodes ensures matching storage pool and network bridge configurations (required for migration)
  • After joining, manage the cluster through the init node's remote
  • VM migration requires matching storage pool names across nodes

Cleanup

# Remove test instances only (keeps VMs and cluster)
./incusos/lab-test --cleanup examples/lab-cluster.yaml

# Destroy VMs only
./incusos/incusos-proxmox --cleanup examples/lab-cluster.yaml

# Full cleanup: VMs + ISOs + seeds + remotes + cache
./incusos/incusos-proxmox --cleanup --deep --yes examples/lab-cluster.yaml

Operations Center Lab

Deploy

./incusos/incusos-proxmox --yes examples/lab-oc.yaml
./incusos/incusos-proxmox --status examples/lab-oc.yaml

Access the Web UI

# Export client certificate for browser
openssl pkcs12 -export \
    -inkey ~/.config/incus/client.key \
    -in ~/.config/incus/client.crt \
    -out client.pfx

# Import client.pfx into Firefox, navigate to https://<OC_IP>:8443

OC CLI Setup

# Copy incus client certs for OC CLI
mkdir -p ~/.config/operations-center
cp ~/.config/incus/client.crt ~/.config/operations-center/
cp ~/.config/incus/client.key ~/.config/operations-center/

# Add OC remote
operations-center remote add lab-oc https://<OC_IP>:8443
operations-center remote switch lab-oc

# Check for provisioning updates
operations-center provisioning update list

Provisioning Workflow

# Create a provisioning token
operations-center provisioning token add --description "lab nodes" --uses 10

# Get provisioned ISO
operations-center provisioning token get-image <token> ~/Downloads/IncusOS-provisioned.iso

# Redeploy nodes with provisioned ISO, or check if manual adoption works

Note: Operations Center is under active development (v0.2.2+). Commands and APIs may change between versions. The --doctor command reports whether the OC CLI is installed.

Troubleshooting

Symptom Likely Cause Fix
SSH connection refused Key auth not set up ssh-copy-id root@<host>
Storage pool not found Wrong pool name in config Check pvesm status on Proxmox
ISO download fails CDN unreachable Download manually, use --iso path/to/file.iso
VM creation fails VMID collision Use higher start_vmid or check qm list
Install times out (10 min) Slow I/O or boot issue Check VM console in Proxmox web UI
No IP reported No guest agent (expected) Check Proxmox console; script uses ARP-based detection
Wrong IP reported Stale ARP entries Script flushes ARP, but verify via Proxmox console
"install media detected" on boot ide2/ide3 still attached Script auto-detaches; if manual, delete ide2+ide3 then start
"no target device matched" Explicit disk-target in seed Remove disk-target; let IncusOS auto-detect
"field server not found" YAML error Wrong preseed structure Use preseed.certificates[] not preseed.server.certificates[]
Cert not auto-detected No Incus keypair Run incus remote list to generate one
genisoimage not found genisoimage not installed Install genisoimage package
Pool not found Resource pool not created Run pveum pool add <name> on Proxmox
Re-run destroys VMs Used option 3 in reconcile Use option 1 (checks) or option 2 (continue) instead
--status shows "not found" VM was cleaned up Deploy first, then check status