30 KiB

Raw Blame History

Testing Guide: incusos-proxmox

A step-by-step guide for deploying IncusOS VMs to Proxmox VE using incusos-proxmox.

Implementation Status

The three scripts (incusos-proxmox, incusos-seed, incusos-iso) implement the full lifecycle:

Config parsing -- YAML with PyYAML or built-in fallback parser
ISO download -- fetches latest IncusOS from the CDN by channel (stable/testing)
Per-VM seed generation -- calls incusos-seed --format iso for each VM
Upload to Proxmox -- SCP (SSH method) or multipart POST (API method)
VM creation -- all IncusOS-required settings enforced (UEFI, TPM, VirtIO-SCSI, etc.)
Automated install monitoring -- polls blockstat.scsi0.wr_bytes via API to detect when disk writes stop (3 stable polls = 15s idle), then stops VM, detaches install media (ide2 + ide3), and boots from disk
Cleanup -- safe teardown with managed-VM marker checks

Cluster Formation (Manual Step)

The cluster and cluster_role fields are accepted in config for documentation purposes, but automatic cluster join is not yet implemented. After deployment, join nodes to the cluster manually:

# On the init node:
incus cluster add <node-name>
# Copy the join token, then on the joining node's preseed or CLI

Authentication

SSH (Default)

Uses key-based SSH as root@<host>. Requires no setup beyond ssh-copy-id.

API Token (Recommended for Production)

Avoids root SSH entirely. Uses a dedicated Proxmox user with minimal privileges.

Minimum Required Privileges

The role below is already the minimum viable set -- every privilege is required by at least one operation in the script:

Privilege	Used For
`VM.Allocate`	Create and destroy VMs
`VM.Config.Disk`	scsi0, efidisk0, tpmstate0
`VM.Config.CPU`	Set cores, cpu=host
`VM.Config.Memory`	Set memory, balloon=0
`VM.Config.Network`	Attach VM to bridge
`VM.Config.CDROM`	Attach ISO (ide2) and seed (ide3)
`VM.Config.Options`	Set boot order, description, agent flag
`VM.Config.HWType`	Set bios=ovmf, machine=q35, scsihw
`VM.PowerMgmt`	Start, stop, and reboot VMs
`VM.Audit`	Query VM status and config
`Datastore.AllocateSpace`	Create disks in storage pool
`Datastore.AllocateTemplate`	Upload ISOs to storage
`Datastore.Audit`	List and check storage pools
`SDN.Use`	Use network bridges

None of these can be removed without breaking functionality.

Setup Commands (Run on Proxmox)

pveum role add IncusOSDeploy --privs "VM.Allocate VM.Config.Disk \
    VM.Config.CPU VM.Config.Memory VM.Config.Network VM.Config.CDROM \
    VM.Config.Options VM.Config.HWType VM.PowerMgmt VM.Audit \
    Datastore.AllocateSpace Datastore.AllocateTemplate \
    Datastore.Audit SDN.Use"

pveum user add automation@pve
pveum aclmod / -user automation@pve -role IncusOSDeploy
pveum user token add automation@pve deploy --privsep 0
# Save the displayed secret!

Optional: Scoped ACLs

For tighter access control, scope the ACL to specific paths instead of /:

pveum aclmod /storage/local -user automation@pve -role IncusOSDeploy
pveum aclmod /storage/local-lvm -user automation@pve -role IncusOSDeploy
pveum aclmod /vms -user automation@pve -role IncusOSDeploy

For a home lab, / (root) is simpler and perfectly acceptable.

Guided Test Scenario

Prerequisites

On your workstation (where you run the script):

# Verify tools
bash --version           # Need 4+
python3 --version        # For YAML parsing
which jq curl ssh scp    # All needed for SSH method
which genisoimage        # Needed for seed ISO images (used by incusos-proxmox)

# Verify the Incus client has generated a keypair
ls ~/.config/incus/client.crt ~/.config/incus/client.key
# If missing, run:  incus remote list   (triggers auto-generation)

On your Proxmox host: nothing to install -- just ensure SSH key auth works.

Step 1: Verify SSH Connectivity

ssh root@<YOUR_PROXMOX_IP> pvesh get /version

You should see JSON with the Proxmox VE version. If this fails, fix SSH key auth first (ssh-copy-id root@<host>).

Step 2: Set Up Proxmox Connection

cp incusos/examples/proxmox.yaml.example incusos/proxmox.yaml

Edit incusos/proxmox.yaml with your Proxmox host settings. This is auto-discovered by the scripts and avoids repeating credentials in each lab config.

Step 3: Create a Test Config

cp incusos/examples/proxmox-minimal.yaml my-test.yaml

Edit my-test.yaml -- change only the host IP (or rely on proxmox.yaml):

proxmox:
  host: <YOUR_PROXMOX_IP>       # <-- change this

defaults:
  start_vmid: 900                # High range to avoid collisions

vms:
  - name: incus-test
    app: incus
    apply_defaults: true
    cores: 4
    memory: 8192
    disk: 64

Using VMID 900+ avoids clashing with existing VMs.

Step 4: Dry Run

./incusos/incusos-proxmox --dry-run my-test.yaml

Verify in the output:

Config parsed correctly (host, node, storage shown)
Client certificate detected (~/.config/incus/client.crt)
Storage pools local-lvm and local found (adjust for your setup)
Network bridge vmbr0 found
VMID 900 allocated for incus-test
ISO version resolved from CDN (e.g., IncusOS_25.03.iso)
[dry-run] prefix on all actions -- nothing actually executed
No errors

Step 5: Deploy (Single VM)

./incusos/incusos-proxmox my-test.yaml

Watch the phases:

Validate -- connects to Proxmox, checks storage/bridge/tools
Confirm -- type y to proceed
Prepare -- downloads IncusOS ISO from CDN (~500 MB), generates seed
Upload -- SCPs ISO + seed to Proxmox
Create -- runs qm create 900 ... with all IncusOS settings
Install -- starts VM, monitors disk I/O via blockstat, detaches media, boots from disk (~2-3 min)

Simultaneously on the Proxmox web UI:

Open https://<YOUR_PROXMOX_IP>:8006
VM 900 (incus-test) should appear
Open its Console tab
Watch the IncusOS installer boot from ISO, read the seed, install to disk, and reboot
After reboot you should see the IncusOS login prompt

Step 6: Check Deployment Status

After the deploy completes, verify with the built-in status command:

./incusos/incusos-proxmox --status my-test.yaml

Verify in the output:

VM incus-test exists and is managed
Status is running, install state is installed
IP address detected
Port 8443 is open
Incus remote status shown (if configured)

Step 7: Verify the Deployed VM

Once the script reports success with an IP address:

# The cert was injected via seed, so no auth prompt appears
incus remote add incus-test <REPORTED_IP> --accept-certificate

# Test connectivity
incus remote list
incus info incus-test:

# If apply_defaults was true, there should be a ZFS pool and bridge
incus storage list incus-test:
incus network list incus-test:

Note: IncusOS is immutable and has no QEMU guest agent. IP detection uses ARP-based lookup (MAC from Proxmox VM config → ARP table). If the IP is wrong, check the Proxmox console — stale ARP entries can be misleading.

Step 8: Test Re-run Reconcile

Re-run the deploy to verify the reconcile menu:

./incusos/incusos-proxmox my-test.yaml

Verify: The script detects the existing VM and shows the interactive menu. Choose option 1 to run status checks without modifying anything.

Also test with --yes to verify it defaults to safe behavior:

./incusos/incusos-proxmox --yes my-test.yaml

Verify: Runs post-deployment checks, does NOT destroy.

Step 9: Clean Up the Test VM

./incusos/incusos-proxmox --cleanup my-test.yaml

Confirms before destroying
Checks the management marker ([incusos-lab:managed] in the VM description)
Stops and destroys with --purge

Remove the remote too:

incus remote remove incus-test

Step 10: (Optional) Test API Method

After single-VM success with SSH, switch to API tokens.

On Proxmox (one-time setup via SSH or web shell):

pveum role add IncusOSDeploy --privs "VM.Allocate VM.Config.Disk \
    VM.Config.CPU VM.Config.Memory VM.Config.Network VM.Config.CDROM \
    VM.Config.Options VM.Config.HWType VM.PowerMgmt VM.Audit \
    Datastore.AllocateSpace Datastore.AllocateTemplate \
    Datastore.Audit SDN.Use"
pveum user add automation@pve
pveum aclmod / -user automation@pve -role IncusOSDeploy
pveum user token add automation@pve deploy --privsep 0
# ^^^ Save the displayed secret!

On your workstation:

export PROXMOX_TOKEN_SECRET="<the-secret-from-above>"

Update my-test.yaml:

proxmox:
  host: <YOUR_PROXMOX_IP>
  method: api
  api_token_id: automation@pve!deploy

Repeat steps 4-7 using the API method.

Step 11: (Optional) Test with Resource Pool

For API method with pool isolation:

On Proxmox:

pveum pool add IncusLab --comment "IncusOS Lab VMs"
pveum pool modify IncusLab --storage local-lvm,local
pveum aclmod /pool/IncusLab -user automation@pve -role IncusOSDeploy

Add to your config:

proxmox:
  pool: IncusLab

Deploy and verify:

./incusos/incusos-proxmox --dry-run my-test.yaml   # Should show pool in plan
./incusos/incusos-proxmox my-test.yaml              # VM created in pool
./incusos/incusos-proxmox --status my-test.yaml     # Status scoped to pool

Step 12: (Optional) Full Lab Deployment

Once single-VM tests pass, deploy the full 4-VM lab:

cp incusos/examples/proxmox-lab.yaml my-lab.yaml
# Edit: change host IP, adjust start_vmid if needed
./incusos/incusos-proxmox --dry-run my-lab.yaml
./incusos/incusos-proxmox my-lab.yaml

This deploys 1 Operations Center + 3 Incus nodes. After deployment:

Access Operations Center via browser at its reported IP
On the init node, run incus cluster add <node-name> to get join tokens
Join the other two nodes using the tokens

Lab Validation with lab-test

The lab-test script automates the end-to-end validation of deployed IncusOS VMs: single-node workloads, cluster formation, migration, and evacuation.

Quick Start: Cluster Testing

# 1. Check environment
./incusos/incusos-proxmox --doctor

# 2. Deploy 3 nodes
./incusos/incusos-proxmox --yes examples/lab-cluster.yaml

# 3. Add incus remotes for each node (use IPs from --status)
./incusos/incusos-proxmox --status examples/lab-cluster.yaml
incus remote add incus-lab-01 <IP1> --accept-certificate
incus remote add incus-lab-02 <IP2> --accept-certificate
incus remote add incus-lab-03 <IP3> --accept-certificate

# 4. Run full lab validation
./incusos/lab-test --yes examples/lab-cluster.yaml

Phase-by-Phase Testing

# Single-node workloads only (container + VM launch/exec)
./incusos/lab-test --phase single --skip-deploy examples/lab-cluster.yaml

# Cluster formation only
./incusos/lab-test --phase cluster --skip-deploy examples/lab-cluster.yaml

# Cluster workloads (scheduler placement + targeted placement)
./incusos/lab-test --phase workload --skip-deploy examples/lab-cluster.yaml

# Migration and evacuation tests
./incusos/lab-test --phase migrate --skip-deploy examples/lab-cluster.yaml

# Extended phases (require a working cluster):
./incusos/lab-test --phase storage --skip-deploy examples/lab-cluster.yaml
./incusos/lab-test --phase network --skip-deploy examples/lab-cluster.yaml
./incusos/lab-test --phase projects --skip-deploy examples/lab-cluster.yaml
./incusos/lab-test --phase backup --skip-deploy examples/lab-cluster.yaml
./incusos/lab-test --phase limits --skip-deploy examples/lab-cluster.yaml
./incusos/lab-test --phase security --skip-deploy examples/lab-cluster.yaml
./incusos/lab-test --phase resilience --skip-deploy examples/lab-cluster.yaml

Extended Test Phases

Phase	Tests	Requires
`storage`	Pool listing, volume create/delete, volume snapshots, pool usage	Single node
`network`	Network listing, bridge create/delete, ACLs, forwards	Single node
`projects`	Project create with limits, custom profiles, profile inheritance	Single node
`backup`	Instance snapshots (create/restore/delete), export/import round-trip	Single node
`limits`	CPU limits, memory limits, disk I/O limits	Single node
`security`	Unprivileged vs privileged containers, nesting, user namespace isolation	Single node
`resilience`	Cluster health, database leader, member state, warnings	3-node cluster

Manual Cluster Formation (Alternative)

If you prefer to form the cluster manually instead of using lab-test, follow the steps below. See also notes/clustering-guide.md for the full reference guide with explanations of what happens under the hood.

Prerequisites

# Check client version (need 6.20+ for remote cluster join)
incus version

# Verify all three remotes are working
incus info incus-lab-01: | head -5
incus info incus-lab-02: | head -5
incus info incus-lab-03: | head -5

Step 1: Set specific IPs on all nodes

IncusOS defaults to core.https_address: :8443 (wildcard). Clustering requires a specific routable IP on each node.

# Discover each node's routable IP
for node in incus-lab-01 incus-lab-02 incus-lab-03; do
  echo "=== $node ==="
  incus query "$node:/1.0" | python3 -c "import sys,json; d=json.load(sys.stdin); \
    [print(a) for a in d['environment']['addresses'] \
     if not a.startswith('10.') and not a.startswith('fd42:') and not a.startswith('[')]"
done

# Set the specific IP on each node (replace with actual IPs)
incus config set incus-lab-01: core.https_address <IP1>:8443
incus config set incus-lab-02: core.https_address <IP2>:8443
incus config set incus-lab-03: core.https_address <IP3>:8443

# Verify connectivity still works after each change
incus info incus-lab-01: | head -3
incus info incus-lab-02: | head -3
incus info incus-lab-03: | head -3

Step 2: Enable clustering on the init node

incus cluster enable incus-lab-01:incus-lab-01

Important: This regenerates the server's TLS certificate. Your remote will break with a certificate error. Fix it:

incus remote switch local          # if incus-lab-01 is the current default
incus remote remove incus-lab-01
incus remote add incus-lab-01 https://<IP1>:8443 --accept-certificate

Verify:

incus cluster list incus-lab-01:
# Should show one member (incus-lab-01), status ONLINE, role database-leader

Step 3: Prepare joining nodes

Nodes with apply_defaults: true have a pre-existing local storage pool and incusbr0 network. These conflict with the cluster join process (the join wizard tries to set member-specific ZFS config keys at cluster level). Delete them before joining:

# For each joining node (incus-lab-02, incus-lab-03):
NODE=incus-lab-02   # then repeat with incus-lab-03

# Remove server-level references to the pool
incus config unset "$NODE": storage.backups_volume
incus config unset "$NODE": storage.images_volume

# Delete the backup and image volumes
incus storage volume delete "$NODE":local backups
incus storage volume delete "$NODE":local images

# Clear default profile references (otherwise pool is "in use")
incus profile device remove "$NODE":default root
incus profile device remove "$NODE":default eth0

# Delete pool and network
incus storage delete "$NODE":local
incus network delete "$NODE":incusbr0

Step 4: Join nodes to the cluster

Interactive (one node at a time):

# Generate join token
incus cluster add incus-lab-01:incus-lab-02

# Join (answer the interactive prompts)
incus cluster join incus-lab-01: incus-lab-02:
# Prompt 1 - IP address: press Enter (accept default)
# Prompt 2 - Member name: press Enter (accept default)
# Prompt 3 - "All existing data is lost": type "yes"
# Prompt 4 - source for storage pool "local": type "local/incus"
# Prompt 5 - zfs.pool_name for storage pool "local": type "local/incus"

Automated (non-interactive):

# Generate token + join in one go
incus cluster add incus-lab-01:incus-lab-02
printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join incus-lab-01: incus-lab-02:

After each join, fix the remote (same certificate issue as the init node):

incus remote remove incus-lab-02
incus remote add incus-lab-02 https://<IP2>:8443 --accept-certificate

Repeat for incus-lab-03.

Step 5: Verify the cluster

# All 3 members should be ONLINE
incus cluster list incus-lab-01:

# Storage pool should show on all members
incus storage show incus-lab-01:local
incus storage show incus-lab-01:local --target incus-lab-01
incus storage show incus-lab-01:local --target incus-lab-02
incus storage show incus-lab-01:local --target incus-lab-03

# All remotes should work
incus info incus-lab-01: | head -3
incus info incus-lab-02: | head -3
incus info incus-lab-03: | head -3

Key points:

apply_defaults: true on all nodes ensures matching storage pool and network bridge names (required for migration) -- but the pools must be deleted on joining nodes before the join
After joining, manage the cluster through the init node's remote
The ZFS pool source is local/incus on IncusOS (auto-created by apply_defaults)
Each join generates a new token (tokens are single-use and time-limited)
VM migration requires matching storage pool names across nodes

Migration Testing (After Cluster Formation)

Once the cluster is formed, validate container and VM migration:

Container migration (stop/move/start)

# Launch on node 2
incus launch images:debian/12 incus-lab-01:test-container --target incus-lab-02
incus exec incus-lab-01:test-container -- bash -c 'echo "hello" > /root/test.txt'

# Migrate to node 3
incus stop incus-lab-01:test-container
incus move incus-lab-01:test-container --target incus-lab-03
incus start incus-lab-01:test-container

# Verify: data survives, processes don't
incus exec incus-lab-01:test-container -- cat /root/test.txt   # "hello"
incus list incus-lab-01: --columns nLs                         # node 3

VM live migration

# Launch and configure for migration
incus launch images:debian/12 incus-lab-01:test-vm --vm --target incus-lab-02
# Wait for VM agent...
incus stop incus-lab-01:test-vm
incus config set incus-lab-01:test-vm migration.stateful=true
incus config set incus-lab-01:test-vm limits.cpu=0-1
incus config device add incus-lab-01:test-vm root disk path=/ pool=local size.state=2GiB
incus start incus-lab-01:test-vm

# Set up heartbeat for state verification
incus exec incus-lab-01:test-vm -- mkdir -p /tmp/migration-test
incus exec incus-lab-01:test-vm -- bash -c \
  'cat > /tmp/migration-test/heartbeat.sh << "EOF"
#!/bin/bash
i=0; while true; do echo $i > /tmp/migration-test/heartbeat; i=$((i+1)); sleep 1; done
EOF
chmod +x /tmp/migration-test/heartbeat.sh'
incus exec incus-lab-01:test-vm -- \
  systemd-run --unit=migration-heartbeat /tmp/migration-test/heartbeat.sh

# Live-migrate (state preserved)
sleep 3
incus exec incus-lab-01:test-vm -- cat /tmp/migration-test/heartbeat   # note value
incus move incus-lab-01:test-vm --target incus-lab-03
sleep 2   # agent needs ~1-2s to reconnect
incus exec incus-lab-01:test-vm -- cat /tmp/migration-test/heartbeat   # should be higher
incus exec incus-lab-01:test-vm -- systemctl is-active migration-heartbeat  # "active"

Cluster evacuation

# Move both workloads to node 2, then evacuate
incus cluster evacuate incus-lab-01:incus-lab-02 --force
incus cluster list incus-lab-01:    # node 2 = EVACUATED
incus list incus-lab-01: --columns nLs  # workloads on other nodes

# Restore
incus cluster restore incus-lab-01:incus-lab-02 --force
incus cluster list incus-lab-01:    # node 2 = ONLINE
incus list incus-lab-01: --columns nLs  # workloads back on node 2

See notes/clustering-guide.md for the full automated migration test script and detailed test results.

Cleanup

# Remove test instances only (keeps VMs and cluster)
./incusos/lab-test --cleanup examples/lab-cluster.yaml

# Destroy VMs only
./incusos/incusos-proxmox --cleanup examples/lab-cluster.yaml

# Full cleanup: VMs + deployment ISO + seeds + remotes + cache
./incusos/incusos-proxmox --cleanup --deep --yes examples/lab-cluster.yaml

# Pool-wide: destroy ALL managed VMs (no config file needed)
./incusos/incusos-proxmox --cleanup-all --yes
./incusos/incusos-proxmox --cleanup-all --deep --yes   # nuke everything

Operations Center Lab (Tested)

Overview

Operations Center (OC) manages IncusOS nodes through a provisioning pipeline: tokens → provisioned ISOs → self-registration → cluster formation.

Hybrid approach (tested, recommended): use incusos-proxmox --iso to deploy nodes from an OC-provisioned ISO. Combines OC auto-registration with incusos-proxmox deployment automation (VM creation, SEED_DATA, install monitoring, media cleanup, IP detection).

Prerequisites

# 1. Install OC CLI
wget https://github.com/FuturFusion/operations-center/releases/download/v0.2.2/operations-center_linux_x86_64
chmod +x operations-center_linux_x86_64
sudo mv operations-center_linux_x86_64 /usr/local/bin/operations-center

# 2. Set up OC client certs
mkdir -p ~/.config/operations-center
cp ~/.config/incus/client.crt ~/.config/operations-center/
cp ~/.config/incus/client.key ~/.config/operations-center/

# 3. Browser cert (for web UI)
openssl pkcs12 -export \
    -inkey ~/.config/incus/client.key \
    -in ~/.config/incus/client.crt \
    -out ~/.config/incus/client.pfx

# 4. Verify
./incusos/incusos-proxmox --doctor

Step 1: Deploy OC Server

./incusos/incusos-proxmox --yes incusos/examples/lab-oc-deploy.yaml
./incusos/incusos-proxmox --status incusos/examples/lab-oc-deploy.yaml
# Note OC_IP from output

Step 2: Connect OC CLI

printf "yes\n" | operations-center remote add oc-lab https://<OC_IP>:8443 --auth-type tls
operations-center remote switch oc-lab

Step 3: Wait for Updates (~3 min)

operations-center provisioning update list
# Wait until at least one shows "ready"

Step 4: Create Token + Seed

# Create token
operations-center provisioning token add --description "Lab cluster" --uses 5 --lifetime 168h
operations-center provisioning token list   # Note UUID

# Create structured pre-seed (MUST use section-level keys)
cat > /tmp/oc-preseed.yaml << 'EOF'
install:
  version: "1"
  force_install: true
  force_reboot: true
EOF

# Attach as named seed
operations-center provisioning token seed add <UUID> proxmox-preseed \
    /tmp/oc-preseed.yaml --description "Force reboot for Proxmox VMs"

Step 5: Download OC-Provisioned ISO

operations-center provisioning token seed get-image <UUID> proxmox-preseed \
    /tmp/IncusOS-oc.iso --type iso --architecture x86_64
ls -lh /tmp/IncusOS-oc.iso   # ~3.4 GB

Step 6: Deploy Nodes (Hybrid)

./incusos/incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes \
    incusos/examples/lab-oc-nodes.yaml
./incusos/incusos-proxmox --status incusos/examples/lab-oc-nodes.yaml

incusos-proxmox creates VMs, uploads the OC ISO to ide2, generates per-node SEED_DATA on ide3 (hostname + force_reboot + app selection + cert), monitors install, removes media, boots from disk, and sets up incus remotes.

Step 7: Verify Auto-Registration

# Nodes register within ~30s of first boot
operations-center provisioning server list
# Should show 3 nodes + OC itself, all "ready"

Step 8: Form Cluster

# Prepare application seed (client cert for incus CLI access)
cat > /tmp/oc-app-config.yaml << EOF
certificates:
  - type: client
    name: lab-client
    certificate: |-
$(sed 's/^/      /' ~/.config/incus/client.crt)
EOF

# Form cluster
operations-center provisioning cluster add oc-cluster \
    https://<NODE_01_IP>:8443 \
    --server-names oc-node-01,oc-node-02,oc-node-03 \
    --server-type incus \
    --application-seed-config /tmp/oc-app-config.yaml

Note: with apply_defaults: true in SEED_DATA, Terraform post-config errors are expected ("already in trust store", "not in pending state"). The cluster forms successfully despite these.

Step 9: Verify

# OC
operations-center provisioning cluster list
operations-center provisioning cluster show oc-cluster

# Incus (re-add remotes after cert change from clustering)
incus remote remove oc-node-01 && incus remote add oc-node-01 https://<IP>:8443 --accept-certificate
incus cluster list oc-node-01:

# Inventory (requires resync)
operations-center provisioning cluster resync oc-cluster
operations-center inventory instance list
operations-center inventory storage-pool list
operations-center inventory network list

Step 10: Test Features

# Workload sync: create via incus, resync, verify in OC
incus launch images:debian/12 oc-node-01:test-ct --target oc-node-01
operations-center provisioning cluster resync oc-cluster
operations-center inventory instance list

# Live migration visibility
incus move oc-node-01:test-vm --target oc-node-03
operations-center provisioning cluster resync oc-cluster
operations-center inventory instance list   # shows new location

# Evacuation
incus cluster evacuate oc-node-01:oc-node-01 --force
operations-center provisioning cluster resync oc-cluster
operations-center inventory instance list   # shows relocated workloads
incus cluster restore oc-node-01:oc-node-01 --force

WARNING: do NOT use operations-center provisioning server system reboot on OC-managed nodes. Guest reboot is safe on standalone IncusOS (tested: simultaneous 3-node reboot, all recover in ~50s with data intact). But on OC-managed nodes, the OC agent runs on boot and fails with errors like "invalid crontab expression", "constraint violation" (re-registration), or Incus hangs at "starting application" (peers in boot loops). The node must be destroyed/redeployed.

Cleanup

# Test workloads
incus delete oc-node-01:test-ct --force 2>/dev/null
incus delete oc-node-01:test-vm --force 2>/dev/null

# Nodes
./incusos/incusos-proxmox --cleanup --deep --yes incusos/examples/lab-oc-nodes.yaml

# OC server
./incusos/incusos-proxmox --cleanup --deep --yes incusos/examples/lab-oc-deploy.yaml

# Remotes
operations-center remote remove oc-lab 2>/dev/null

Note: Operations Center is under active development (v0.2.2+). See notes/operations-center-guide.md for the full tested reference.

Troubleshooting

Symptom	Likely Cause	Fix
SSH connection refused	Key auth not set up	`ssh-copy-id root@<host>`
Storage pool not found	Wrong pool name in config	Check `pvesm status` on Proxmox
ISO download fails	CDN unreachable	Download manually, use `--iso path/to/file.iso`
VM creation fails	VMID collision	Use higher `start_vmid` or check `qm list`
Install times out (10 min)	Slow I/O or boot issue	Check VM console in Proxmox web UI
No IP reported	No guest agent (expected)	Check Proxmox console; script uses ARP-based detection
Wrong IP reported	Stale ARP entries	Script flushes ARP, but verify via Proxmox console
"install media detected" on boot	ide2/ide3 still attached	Script auto-detaches; if manual, delete ide2+ide3 then start
"no target device matched"	Explicit `disk-target` in seed	Remove `disk-target`; let IncusOS auto-detect
"field server not found" YAML error	Wrong preseed structure	Use `preseed.certificates[]` not `preseed.server.certificates[]`
Cert not auto-detected	No Incus keypair	Run `incus remote list` to generate one
`genisoimage` not found	genisoimage not installed	Install `genisoimage` package
Pool not found	Resource pool not created	Run `pveum pool add <name>` on Proxmox
Re-run destroys VMs	Used option 3 in reconcile	Use option 1 (checks) or option 2 (continue) instead
`--status` shows "not found"	VM was cleaned up	Deploy first, then check status
VM live migration fails: "Missing section footer for ICH9LPC"	`limits.cpu` not set or set as integer	Use range syntax: `limits.cpu=0-1` (not `2`); see `notes/clustering-guide.md`
`migration.stateful` cannot be set while running	Incus requires VM to be stopped	Stop VM first, set config, then start
`certificate is valid for 127.0.0.1, ::1, not <IP>`	Cluster cert regenerated on enable/join	Remove and re-add remote with `--accept-certificate`
`Config key "zfs.pool_name" is cluster member specific`	Storage pool exists on joining node	Delete pool/network before joining; see clustering guide
OC `get-image` fails	No updates in `ready` state yet	Wait ~3 min for OC to download packages; check `provisioning update list`
Node doesn't appear in OC server list	ISO not OC-provisioned or token exhausted	Verify ISO from `get-image` with valid token; check `token list` for remaining uses
OC web UI returns 403	Client cert not imported in browser	Import `client.pfx` into browser certificates
Bash mangles Proxmox API token with `!`	History expansion	Use single quotes around token value or store in variable
OC token seed fields all `{}`	Flat YAML format used	Use structured format: `install: { force_reboot: true }` not `force_reboot: true` at root
OC cluster Terraform errors ("already in trust store")	`apply_defaults: true` pre-created resources	Non-fatal -- cluster forms despite errors. Skip `--application-seed-config` or skip `apply_defaults`
OC reboot kills OC-managed node (port 8443 closed)	OC agent boot failures: invalid crontab, re-registration conflict, or peers in boot loops (standalone reboot is safe)	Destroy and redeploy OC-managed VM. Guest reboot works fine on standalone IncusOS
OC inventory empty after creating workloads	Inventory not real-time	Run `operations-center provisioning cluster resync <name>`
OC shows stale server entry after node replacement	Out-of-band cluster change	OC can't detect `incus cluster remove`; stale entries persist
OC `server remove` fails ("part of cluster")	Server is in OC cluster	Must remove cluster first or handle via OC cluster management
`cluster join` prompts for `tunnel.mesh.interface`	OC adds `meshbr0` network	Add extra `\n` for meshbr0 prompt: `printf '\n\nyes\n\nlocal/incus\nlocal/incus\n'`
VMID collision (500/403 on VM create)	Pool-scoped API token can't see VMs outside pool	Use `start_vmid: 400` or higher in config to avoid collisions

30 KiB Raw Blame History

Testing Guide: incusos-proxmox

Implementation Status

Cluster Formation (Manual Step)

Authentication

SSH (Default)

API Token (Recommended for Production)

Minimum Required Privileges

Setup Commands (Run on Proxmox)

Optional: Scoped ACLs

Guided Test Scenario

Prerequisites

Step 1: Verify SSH Connectivity

Step 2: Set Up Proxmox Connection

Step 3: Create a Test Config

Step 4: Dry Run

Step 5: Deploy (Single VM)

Step 6: Check Deployment Status

Step 7: Verify the Deployed VM

Step 8: Test Re-run Reconcile

Step 9: Clean Up the Test VM

Step 10: (Optional) Test API Method

Step 11: (Optional) Test with Resource Pool

Step 12: (Optional) Full Lab Deployment

Lab Validation with lab-test

Quick Start: Cluster Testing

Phase-by-Phase Testing

Extended Test Phases

Manual Cluster Formation (Alternative)

Prerequisites

Step 1: Set specific IPs on all nodes

Step 2: Enable clustering on the init node

Step 3: Prepare joining nodes

Step 4: Join nodes to the cluster

Step 5: Verify the cluster

Migration Testing (After Cluster Formation)

Container migration (stop/move/start)

VM live migration

Cluster evacuation

Cleanup

Operations Center Lab (Tested)

Overview

Prerequisites

Step 1: Deploy OC Server

Step 2: Connect OC CLI

Step 3: Wait for Updates (~3 min)

Step 4: Create Token + Seed

Step 5: Download OC-Provisioned ISO

Step 6: Deploy Nodes (Hybrid)

Step 7: Verify Auto-Registration

Step 8: Form Cluster

Step 9: Verify

Step 10: Test Features

Cleanup

Troubleshooting

30 KiB

Raw Blame History