incus-contrib/incusos/TESTING.md

# Testing Guide: incusos-proxmox

A step-by-step guide for deploying IncusOS VMs to Proxmox VE using
`incusos-proxmox`.

## Implementation Status

The three scripts (`incusos-proxmox`, `incusos-seed`, `incusos-iso`) implement
the full lifecycle:

- **Config parsing** -- YAML with PyYAML or built-in fallback parser
- **ISO download** -- fetches latest IncusOS from the CDN by channel
  (stable/testing)
- **Per-VM seed generation** -- calls `incusos-seed --format iso` for each VM
- **Upload to Proxmox** -- SCP (SSH method) or multipart POST (API method)
- **VM creation** -- all IncusOS-required settings enforced (UEFI, TPM,
  VirtIO-SCSI, etc.)
- **Automated install monitoring** -- polls `blockstat.scsi0.wr_bytes` via API
  to detect when disk writes stop (3 stable polls = 15s idle), then stops VM,
  detaches install media (ide2 + ide3), and boots from disk
- **Cleanup** -- safe teardown with managed-VM marker checks

### Cluster Formation (Manual Step)

The `cluster` and `cluster_role` fields are accepted in config for
documentation purposes, but automatic cluster join is not yet implemented.
After deployment, join nodes to the cluster manually:

```bash
# On the init node:
incus cluster add <node-name>
# Copy the join token, then on the joining node's preseed or CLI
```

---

## Authentication

### SSH (Default)

Uses key-based SSH as `root@<host>`. Requires no setup beyond
`ssh-copy-id`.

### API Token (Recommended for Production)

Avoids root SSH entirely. Uses a dedicated Proxmox user with minimal
privileges.

#### Minimum Required Privileges

The role below is already the minimum viable set -- every privilege is
required by at least one operation in the script:

| Privilege | Used For |
|-----------|----------|
| `VM.Allocate` | Create and destroy VMs |
| `VM.Config.Disk` | scsi0, efidisk0, tpmstate0 |
| `VM.Config.CPU` | Set cores, cpu=host |
| `VM.Config.Memory` | Set memory, balloon=0 |
| `VM.Config.Network` | Attach VM to bridge |
| `VM.Config.CDROM` | Attach ISO (ide2) and seed (ide3) |
| `VM.Config.Options` | Set boot order, description, agent flag |
| `VM.Config.HWType` | Set bios=ovmf, machine=q35, scsihw |
| `VM.PowerMgmt` | Start, stop, and reboot VMs |
| `VM.Audit` | Query VM status and config |
| `Datastore.AllocateSpace` | Create disks in storage pool |
| `Datastore.AllocateTemplate` | Upload ISOs to storage |
| `Datastore.Audit` | List and check storage pools |
| `SDN.Use` | Use network bridges |

None of these can be removed without breaking functionality.

#### Setup Commands (Run on Proxmox)

```bash
pveum role add IncusOSDeploy --privs "VM.Allocate VM.Config.Disk \
    VM.Config.CPU VM.Config.Memory VM.Config.Network VM.Config.CDROM \
    VM.Config.Options VM.Config.HWType VM.PowerMgmt VM.Audit \
    Datastore.AllocateSpace Datastore.AllocateTemplate \
    Datastore.Audit SDN.Use"

pveum user add automation@pve
pveum aclmod / -user automation@pve -role IncusOSDeploy
pveum user token add automation@pve deploy --privsep 0
# Save the displayed secret!
```

#### Optional: Scoped ACLs

For tighter access control, scope the ACL to specific paths instead of `/`:

```bash
pveum aclmod /storage/local -user automation@pve -role IncusOSDeploy
pveum aclmod /storage/local-lvm -user automation@pve -role IncusOSDeploy
pveum aclmod /vms -user automation@pve -role IncusOSDeploy
```

For a home lab, `/` (root) is simpler and perfectly acceptable.

---

## Guided Test Scenario

### Prerequisites

**On your workstation** (where you run the script):

```bash
# Verify tools
bash --version           # Need 4+
python3 --version        # For YAML parsing
which jq curl ssh scp    # All needed for SSH method
which genisoimage        # Needed for seed ISO images (used by incusos-proxmox)

# Verify the Incus client has generated a keypair
ls ~/.config/incus/client.crt ~/.config/incus/client.key
# If missing, run:  incus remote list   (triggers auto-generation)
```

**On your Proxmox host**: nothing to install -- just ensure SSH key auth
works.

### Step 1: Verify SSH Connectivity

```bash
ssh root@<YOUR_PROXMOX_IP> pvesh get /version
```

You should see JSON with the Proxmox VE version. If this fails, fix SSH
key auth first (`ssh-copy-id root@<host>`).

### Step 2: Set Up Proxmox Connection

```bash
cp incusos/examples/proxmox.yaml.example incusos/proxmox.yaml
```

Edit `incusos/proxmox.yaml` with your Proxmox host settings. This is
auto-discovered by the scripts and avoids repeating credentials in each
lab config.

### Step 3: Create a Test Config

```bash
cp incusos/examples/proxmox-minimal.yaml my-test.yaml
```

Edit `my-test.yaml` -- change only the host IP (or rely on proxmox.yaml):

```yaml
proxmox:
  host: <YOUR_PROXMOX_IP>       # <-- change this

defaults:
  start_vmid: 900                # High range to avoid collisions

vms:
  - name: incus-test
    app: incus
    apply_defaults: true
    cores: 4
    memory: 8192
    disk: 64
```

Using VMID 900+ avoids clashing with existing VMs.

### Step 4: Dry Run

```bash
./incusos/incusos-proxmox --dry-run my-test.yaml
```

**Verify in the output:**

- Config parsed correctly (host, node, storage shown)
- Client certificate detected (`~/.config/incus/client.crt`)
- Storage pools `local-lvm` and `local` found (adjust for your setup)
- Network bridge `vmbr0` found
- VMID 900 allocated for `incus-test`
- ISO version resolved from CDN (e.g., `IncusOS_25.03.iso`)
- `[dry-run]` prefix on all actions -- nothing actually executed
- No errors

### Step 5: Deploy (Single VM)

```bash
./incusos/incusos-proxmox my-test.yaml
```

**Watch the phases:**

1. **Validate** -- connects to Proxmox, checks storage/bridge/tools
2. **Confirm** -- type `y` to proceed
3. **Prepare** -- downloads IncusOS ISO from CDN (~500 MB), generates seed
4. **Upload** -- SCPs ISO + seed to Proxmox
5. **Create** -- runs `qm create 900 ...` with all IncusOS settings
6. **Install** -- starts VM, monitors disk I/O via blockstat, detaches media, boots from disk (~2-3 min)

**Simultaneously on the Proxmox web UI:**

- Open `https://<YOUR_PROXMOX_IP>:8006`
- VM 900 (`incus-test`) should appear
- Open its Console tab
- Watch the IncusOS installer boot from ISO, read the seed, install to
  disk, and reboot
- After reboot you should see the IncusOS login prompt

### Step 6: Check Deployment Status

After the deploy completes, verify with the built-in status command:

```bash
./incusos/incusos-proxmox --status my-test.yaml
```

**Verify in the output:**

- VM `incus-test` exists and is managed
- Status is `running`, install state is `installed`
- IP address detected
- Port 8443 is open
- Incus remote status shown (if configured)

### Step 7: Verify the Deployed VM

Once the script reports success with an IP address:

```bash
# The cert was injected via seed, so no auth prompt appears
incus remote add incus-test <REPORTED_IP> --accept-certificate

# Test connectivity
incus remote list
incus info incus-test:

# If apply_defaults was true, there should be a ZFS pool and bridge
incus storage list incus-test:
incus network list incus-test:
```

**Note:** IncusOS is immutable and has no QEMU guest agent. IP detection uses
ARP-based lookup (MAC from Proxmox VM config → ARP table). If the IP is wrong,
check the Proxmox console — stale ARP entries can be misleading.

### Step 8: Test Re-run Reconcile

Re-run the deploy to verify the reconcile menu:

```bash
./incusos/incusos-proxmox my-test.yaml
```

**Verify:** The script detects the existing VM and shows the interactive menu.
Choose option 1 to run status checks without modifying anything.

Also test with `--yes` to verify it defaults to safe behavior:

```bash
./incusos/incusos-proxmox --yes my-test.yaml
```

**Verify:** Runs post-deployment checks, does NOT destroy.

### Step 9: Clean Up the Test VM

```bash
./incusos/incusos-proxmox --cleanup my-test.yaml
```

- Confirms before destroying
- Checks the management marker (`[incusos-lab:managed]` in the VM
  description)
- Stops and destroys with `--purge`

Remove the remote too:

```bash
incus remote remove incus-test
```

### Step 10: (Optional) Test API Method

After single-VM success with SSH, switch to API tokens.

**On Proxmox** (one-time setup via SSH or web shell):

```bash
pveum role add IncusOSDeploy --privs "VM.Allocate VM.Config.Disk \
    VM.Config.CPU VM.Config.Memory VM.Config.Network VM.Config.CDROM \
    VM.Config.Options VM.Config.HWType VM.PowerMgmt VM.Audit \
    Datastore.AllocateSpace Datastore.AllocateTemplate \
    Datastore.Audit SDN.Use"
pveum user add automation@pve
pveum aclmod / -user automation@pve -role IncusOSDeploy
pveum user token add automation@pve deploy --privsep 0
# ^^^ Save the displayed secret!
```

**On your workstation:**

```bash
export PROXMOX_TOKEN_SECRET="<the-secret-from-above>"
```

Update `my-test.yaml`:

```yaml
proxmox:
  host: <YOUR_PROXMOX_IP>
  method: api
  api_token_id: automation@pve!deploy
```

Repeat steps 4-7 using the API method.

### Step 11: (Optional) Test with Resource Pool

For API method with pool isolation:

**On Proxmox:**

```bash
pveum pool add IncusLab --comment "IncusOS Lab VMs"
pveum pool modify IncusLab --storage local-lvm,local
pveum aclmod /pool/IncusLab -user automation@pve -role IncusOSDeploy
```

Add to your config:

```yaml
proxmox:
  pool: IncusLab
```

Deploy and verify:

```bash
./incusos/incusos-proxmox --dry-run my-test.yaml   # Should show pool in plan
./incusos/incusos-proxmox my-test.yaml              # VM created in pool
./incusos/incusos-proxmox --status my-test.yaml     # Status scoped to pool
```

### Step 12: (Optional) Full Lab Deployment

Once single-VM tests pass, deploy the full 4-VM lab:

```bash
cp incusos/examples/proxmox-lab.yaml my-lab.yaml
# Edit: change host IP, adjust start_vmid if needed
./incusos/incusos-proxmox --dry-run my-lab.yaml
./incusos/incusos-proxmox my-lab.yaml
```

This deploys 1 Operations Center + 3 Incus nodes. After deployment:

1. Access Operations Center via browser at its reported IP
2. On the init node, run `incus cluster add <node-name>` to get join tokens
3. Join the other two nodes using the tokens

---

## Lab Validation with lab-test

The `lab-test` script automates the end-to-end validation of deployed IncusOS
VMs: single-node workloads, cluster formation, migration, and evacuation.

### Quick Start: Cluster Testing

```bash
# 1. Check environment
./incusos/incusos-proxmox --doctor

# 2. Deploy 3 nodes
./incusos/incusos-proxmox --yes examples/lab-cluster.yaml

# 3. Add incus remotes for each node (use IPs from --status)
./incusos/incusos-proxmox --status examples/lab-cluster.yaml
incus remote add incus-lab-01 <IP1> --accept-certificate
incus remote add incus-lab-02 <IP2> --accept-certificate
incus remote add incus-lab-03 <IP3> --accept-certificate

# 4. Run full lab validation
./incusos/lab-test --yes examples/lab-cluster.yaml
```

### Phase-by-Phase Testing

```bash
# Single-node workloads only (container + VM launch/exec)
./incusos/lab-test --phase single --skip-deploy examples/lab-cluster.yaml

# Cluster formation only
./incusos/lab-test --phase cluster --skip-deploy examples/lab-cluster.yaml

# Cluster workloads (scheduler placement + targeted placement)
./incusos/lab-test --phase workload --skip-deploy examples/lab-cluster.yaml

# Migration and evacuation tests
./incusos/lab-test --phase migrate --skip-deploy examples/lab-cluster.yaml

# Extended phases (require a working cluster):
./incusos/lab-test --phase storage --skip-deploy examples/lab-cluster.yaml
./incusos/lab-test --phase network --skip-deploy examples/lab-cluster.yaml
./incusos/lab-test --phase projects --skip-deploy examples/lab-cluster.yaml
./incusos/lab-test --phase backup --skip-deploy examples/lab-cluster.yaml
./incusos/lab-test --phase limits --skip-deploy examples/lab-cluster.yaml
./incusos/lab-test --phase security --skip-deploy examples/lab-cluster.yaml
./incusos/lab-test --phase resilience --skip-deploy examples/lab-cluster.yaml
```

### Extended Test Phases

| Phase | Tests | Requires |
|-------|-------|----------|
| `storage` | Pool listing, volume create/delete, volume snapshots, pool usage | Single node |
| `network` | Network listing, bridge create/delete, ACLs, forwards | Single node |
| `projects` | Project create with limits, custom profiles, profile inheritance | Single node |
| `backup` | Instance snapshots (create/restore/delete), export/import round-trip | Single node |
| `limits` | CPU limits, memory limits, disk I/O limits | Single node |
| `security` | Unprivileged vs privileged containers, nesting, user namespace isolation | Single node |
| `resilience` | Cluster health, database leader, member state, warnings | 3-node cluster |

### Manual Cluster Formation (Alternative)

If you prefer to form the cluster manually instead of using `lab-test`,
follow the steps below. See also `notes/clustering-guide.md` for the full
reference guide with explanations of what happens under the hood.

#### Prerequisites

```bash
# Check client version (need 6.20+ for remote cluster join)
incus version

# Verify all three remotes are working
incus info incus-lab-01: | head -5
incus info incus-lab-02: | head -5
incus info incus-lab-03: | head -5
```

#### Step 1: Set specific IPs on all nodes

IncusOS defaults to `core.https_address: :8443` (wildcard). Clustering
requires a specific routable IP on each node.

```bash
# Discover each node's routable IP
for node in incus-lab-01 incus-lab-02 incus-lab-03; do
  echo "=== $node ==="
  incus query "$node:/1.0" | python3 -c "import sys,json; d=json.load(sys.stdin); \
    [print(a) for a in d['environment']['addresses'] \
     if not a.startswith('10.') and not a.startswith('fd42:') and not a.startswith('[')]"
done

# Set the specific IP on each node (replace with actual IPs)
incus config set incus-lab-01: core.https_address <IP1>:8443
incus config set incus-lab-02: core.https_address <IP2>:8443
incus config set incus-lab-03: core.https_address <IP3>:8443

# Verify connectivity still works after each change
incus info incus-lab-01: | head -3
incus info incus-lab-02: | head -3
incus info incus-lab-03: | head -3
```

#### Step 2: Enable clustering on the init node

```bash
incus cluster enable incus-lab-01:incus-lab-01
```

> **Important**: This regenerates the server's TLS certificate. Your remote
> will break with a certificate error. Fix it:

```bash
incus remote switch local          # if incus-lab-01 is the current default
incus remote remove incus-lab-01
incus remote add incus-lab-01 https://<IP1>:8443 --accept-certificate
```

Verify:

```bash
incus cluster list incus-lab-01:
# Should show one member (incus-lab-01), status ONLINE, role database-leader
```

#### Step 3: Prepare joining nodes

Nodes with `apply_defaults: true` have a pre-existing `local` storage pool
and `incusbr0` network. These conflict with the cluster join process (the
join wizard tries to set member-specific ZFS config keys at cluster level).
Delete them before joining:

```bash
# For each joining node (incus-lab-02, incus-lab-03):
NODE=incus-lab-02   # then repeat with incus-lab-03

# Remove server-level references to the pool
incus config unset "$NODE": storage.backups_volume
incus config unset "$NODE": storage.images_volume

# Delete the backup and image volumes
incus storage volume delete "$NODE":local backups
incus storage volume delete "$NODE":local images

# Clear default profile references (otherwise pool is "in use")
incus profile device remove "$NODE":default root
incus profile device remove "$NODE":default eth0

# Delete pool and network
incus storage delete "$NODE":local
incus network delete "$NODE":incusbr0
```

#### Step 4: Join nodes to the cluster

**Interactive** (one node at a time):

```bash
# Generate join token
incus cluster add incus-lab-01:incus-lab-02

# Join (answer the interactive prompts)
incus cluster join incus-lab-01: incus-lab-02:
# Prompt 1 - IP address: press Enter (accept default)
# Prompt 2 - Member name: press Enter (accept default)
# Prompt 3 - "All existing data is lost": type "yes"
# Prompt 4 - source for storage pool "local": type "local/incus"
# Prompt 5 - zfs.pool_name for storage pool "local": type "local/incus"
```

**Automated** (non-interactive):

```bash
# Generate token + join in one go
incus cluster add incus-lab-01:incus-lab-02
printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join incus-lab-01: incus-lab-02:
```

After each join, fix the remote (same certificate issue as the init node):

```bash
incus remote remove incus-lab-02
incus remote add incus-lab-02 https://<IP2>:8443 --accept-certificate
```

Repeat for incus-lab-03.

#### Step 5: Verify the cluster

```bash
# All 3 members should be ONLINE
incus cluster list incus-lab-01:

# Storage pool should show on all members
incus storage show incus-lab-01:local
incus storage show incus-lab-01:local --target incus-lab-01
incus storage show incus-lab-01:local --target incus-lab-02
incus storage show incus-lab-01:local --target incus-lab-03

# All remotes should work
incus info incus-lab-01: | head -3
incus info incus-lab-02: | head -3
incus info incus-lab-03: | head -3
```

**Key points:**
- `apply_defaults: true` on all nodes ensures matching storage pool and
  network bridge names (required for migration) -- but the pools must be
  deleted on joining nodes before the join
- After joining, manage the cluster through the init node's remote
- The ZFS pool source is `local/incus` on IncusOS (auto-created by
  `apply_defaults`)
- Each join generates a new token (tokens are single-use and time-limited)
- VM migration requires matching storage pool names across nodes

### Migration Testing (After Cluster Formation)

Once the cluster is formed, validate container and VM migration:

#### Container migration (stop/move/start)

```bash
# Launch on node 2
incus launch images:debian/12 incus-lab-01:test-container --target incus-lab-02
incus exec incus-lab-01:test-container -- bash -c 'echo "hello" > /root/test.txt'

# Migrate to node 3
incus stop incus-lab-01:test-container
incus move incus-lab-01:test-container --target incus-lab-03
incus start incus-lab-01:test-container

# Verify: data survives, processes don't
incus exec incus-lab-01:test-container -- cat /root/test.txt   # "hello"
incus list incus-lab-01: --columns nLs                         # node 3
```

#### VM live migration

```bash
# Launch and configure for migration
incus launch images:debian/12 incus-lab-01:test-vm --vm --target incus-lab-02
# Wait for VM agent...
incus stop incus-lab-01:test-vm
incus config set incus-lab-01:test-vm migration.stateful=true
incus config set incus-lab-01:test-vm limits.cpu=0-1
incus config device add incus-lab-01:test-vm root disk path=/ pool=local size.state=2GiB
incus start incus-lab-01:test-vm

# Set up heartbeat for state verification
incus exec incus-lab-01:test-vm -- mkdir -p /tmp/migration-test
incus exec incus-lab-01:test-vm -- bash -c \
  'cat > /tmp/migration-test/heartbeat.sh << "EOF"
#!/bin/bash
i=0; while true; do echo $i > /tmp/migration-test/heartbeat; i=$((i+1)); sleep 1; done
EOF
chmod +x /tmp/migration-test/heartbeat.sh'
incus exec incus-lab-01:test-vm -- \
  systemd-run --unit=migration-heartbeat /tmp/migration-test/heartbeat.sh

# Live-migrate (state preserved)
sleep 3
incus exec incus-lab-01:test-vm -- cat /tmp/migration-test/heartbeat   # note value
incus move incus-lab-01:test-vm --target incus-lab-03
sleep 2   # agent needs ~1-2s to reconnect
incus exec incus-lab-01:test-vm -- cat /tmp/migration-test/heartbeat   # should be higher
incus exec incus-lab-01:test-vm -- systemctl is-active migration-heartbeat  # "active"
```

#### Cluster evacuation

```bash
# Move both workloads to node 2, then evacuate
incus cluster evacuate incus-lab-01:incus-lab-02 --force
incus cluster list incus-lab-01:    # node 2 = EVACUATED
incus list incus-lab-01: --columns nLs  # workloads on other nodes

# Restore
incus cluster restore incus-lab-01:incus-lab-02 --force
incus cluster list incus-lab-01:    # node 2 = ONLINE
incus list incus-lab-01: --columns nLs  # workloads back on node 2
```

See `notes/clustering-guide.md` for the full automated migration test script
and detailed test results.

### Cleanup

```bash
# Remove test instances only (keeps VMs and cluster)
./incusos/lab-test --cleanup examples/lab-cluster.yaml

# Destroy VMs only
./incusos/incusos-proxmox --cleanup examples/lab-cluster.yaml

# Full cleanup: VMs + deployment ISO + seeds + remotes + cache
./incusos/incusos-proxmox --cleanup --deep --yes examples/lab-cluster.yaml

# Pool-wide: destroy ALL managed VMs (no config file needed)
./incusos/incusos-proxmox --cleanup-all --yes
./incusos/incusos-proxmox --cleanup-all --deep --yes   # nuke everything
```

---

## Operations Center Lab (Tested)

### Overview

Operations Center (OC) manages IncusOS nodes through a provisioning pipeline:
tokens → provisioned ISOs → self-registration → cluster formation.

**Hybrid approach** (tested, recommended): use `incusos-proxmox --iso` to
deploy nodes from an OC-provisioned ISO. Combines OC auto-registration with
`incusos-proxmox` deployment automation (VM creation, SEED_DATA, install
monitoring, media cleanup, IP detection).

### Prerequisites

```bash
# 1. Install OC CLI
wget https://github.com/FuturFusion/operations-center/releases/download/v0.2.2/operations-center_linux_x86_64
chmod +x operations-center_linux_x86_64
sudo mv operations-center_linux_x86_64 /usr/local/bin/operations-center

# 2. Set up OC client certs
mkdir -p ~/.config/operations-center
cp ~/.config/incus/client.crt ~/.config/operations-center/
cp ~/.config/incus/client.key ~/.config/operations-center/

# 3. Browser cert (for web UI)
openssl pkcs12 -export \
    -inkey ~/.config/incus/client.key \
    -in ~/.config/incus/client.crt \
    -out ~/.config/incus/client.pfx

# 4. Verify
./incusos/incusos-proxmox --doctor
```

### Step 1: Deploy OC Server

```bash
./incusos/incusos-proxmox --yes incusos/examples/lab-oc-deploy.yaml
./incusos/incusos-proxmox --status incusos/examples/lab-oc-deploy.yaml
# Note OC_IP from output
```

### Step 2: Connect OC CLI

```bash
printf "yes\n" | operations-center remote add oc-lab https://<OC_IP>:8443 --auth-type tls
operations-center remote switch oc-lab
```

### Step 3: Wait for Updates (~3 min)

```bash
operations-center provisioning update list
# Wait until at least one shows "ready"
```

### Step 4: Create Token + Seed

```bash
# Create token
operations-center provisioning token add --description "Lab cluster" --uses 5 --lifetime 168h
operations-center provisioning token list   # Note UUID

# Create structured pre-seed (MUST use section-level keys)
cat > /tmp/oc-preseed.yaml << 'EOF'
install:
  version: "1"
  force_install: true
  force_reboot: true
EOF

# Attach as named seed
operations-center provisioning token seed add <UUID> proxmox-preseed \
    /tmp/oc-preseed.yaml --description "Force reboot for Proxmox VMs"
```

### Step 5: Download OC-Provisioned ISO

```bash
operations-center provisioning token seed get-image <UUID> proxmox-preseed \
    /tmp/IncusOS-oc.iso --type iso --architecture x86_64
ls -lh /tmp/IncusOS-oc.iso   # ~3.4 GB
```

### Step 6: Deploy Nodes (Hybrid)

```bash
./incusos/incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes \
    incusos/examples/lab-oc-nodes.yaml
./incusos/incusos-proxmox --status incusos/examples/lab-oc-nodes.yaml
```

`incusos-proxmox` creates VMs, uploads the OC ISO to ide2, generates per-node
SEED_DATA on ide3 (hostname + force_reboot + app selection + cert), monitors
install, removes media, boots from disk, and sets up incus remotes.

### Step 7: Verify Auto-Registration

```bash
# Nodes register within ~30s of first boot
operations-center provisioning server list
# Should show 3 nodes + OC itself, all "ready"
```

### Step 8: Form Cluster

```bash
# Prepare application seed (client cert for incus CLI access)
cat > /tmp/oc-app-config.yaml << EOF
certificates:
  - type: client
    name: lab-client
    certificate: |-
$(sed 's/^/      /' ~/.config/incus/client.crt)
EOF

# Form cluster
operations-center provisioning cluster add oc-cluster \
    https://<NODE_01_IP>:8443 \
    --server-names oc-node-01,oc-node-02,oc-node-03 \
    --server-type incus \
    --application-seed-config /tmp/oc-app-config.yaml
```

**Note**: with `apply_defaults: true` in SEED_DATA, Terraform post-config
errors are expected ("already in trust store", "not in pending state"). The
cluster forms successfully despite these.

### Step 9: Verify

```bash
# OC
operations-center provisioning cluster list
operations-center provisioning cluster show oc-cluster

# Incus (re-add remotes after cert change from clustering)
incus remote remove oc-node-01 && incus remote add oc-node-01 https://<IP>:8443 --accept-certificate
incus cluster list oc-node-01:

# Inventory (requires resync)
operations-center provisioning cluster resync oc-cluster
operations-center inventory instance list
operations-center inventory storage-pool list
operations-center inventory network list
```

### Step 10: Test Features

```bash
# Workload sync: create via incus, resync, verify in OC
incus launch images:debian/12 oc-node-01:test-ct --target oc-node-01
operations-center provisioning cluster resync oc-cluster
operations-center inventory instance list

# Live migration visibility
incus move oc-node-01:test-vm --target oc-node-03
operations-center provisioning cluster resync oc-cluster
operations-center inventory instance list   # shows new location

# Evacuation
incus cluster evacuate oc-node-01:oc-node-01 --force
operations-center provisioning cluster resync oc-cluster
operations-center inventory instance list   # shows relocated workloads
incus cluster restore oc-node-01:oc-node-01 --force
```

**WARNING**: do NOT use `operations-center provisioning server system reboot`
on OC-managed nodes. Guest reboot is safe on standalone IncusOS (tested:
simultaneous 3-node reboot, all recover in ~50s with data intact). But on
OC-managed nodes, the OC agent runs on boot and fails with errors like
"invalid crontab expression", "constraint violation" (re-registration), or
Incus hangs at "starting application" (peers in boot loops). The node must
be destroyed/redeployed.

### Cleanup

```bash
# Test workloads
incus delete oc-node-01:test-ct --force 2>/dev/null
incus delete oc-node-01:test-vm --force 2>/dev/null

# Nodes
./incusos/incusos-proxmox --cleanup --deep --yes incusos/examples/lab-oc-nodes.yaml

# OC server
./incusos/incusos-proxmox --cleanup --deep --yes incusos/examples/lab-oc-deploy.yaml

# Remotes
operations-center remote remove oc-lab 2>/dev/null
```

**Note:** Operations Center is under active development (v0.2.2+). See
`notes/operations-center-guide.md` for the full tested reference.

---

## Troubleshooting

| Symptom | Likely Cause | Fix |
|---------|-------------|-----|
| SSH connection refused | Key auth not set up | `ssh-copy-id root@<host>` |
| Storage pool not found | Wrong pool name in config | Check `pvesm status` on Proxmox |
| ISO download fails | CDN unreachable | Download manually, use `--iso path/to/file.iso` |
| VM creation fails | VMID collision | Use higher `start_vmid` or check `qm list` |
| Install times out (10 min) | Slow I/O or boot issue | Check VM console in Proxmox web UI |
| No IP reported | No guest agent (expected) | Check Proxmox console; script uses ARP-based detection |
| Wrong IP reported | Stale ARP entries | Script flushes ARP, but verify via Proxmox console |
| "install media detected" on boot | ide2/ide3 still attached | Script auto-detaches; if manual, delete ide2+ide3 then start |
| "no target device matched" | Explicit `disk-target` in seed | Remove `disk-target`; let IncusOS auto-detect |
| "field server not found" YAML error | Wrong preseed structure | Use `preseed.certificates[]` not `preseed.server.certificates[]` |
| Cert not auto-detected | No Incus keypair | Run `incus remote list` to generate one |
| `genisoimage` not found | genisoimage not installed | Install `genisoimage` package |
| Pool not found | Resource pool not created | Run `pveum pool add <name>` on Proxmox |
| Re-run destroys VMs | Used option 3 in reconcile | Use option 1 (checks) or option 2 (continue) instead |
| `--status` shows "not found" | VM was cleaned up | Deploy first, then check status |
| VM live migration fails: "Missing section footer for ICH9LPC" | `limits.cpu` not set or set as integer | Use range syntax: `limits.cpu=0-1` (not `2`); see `notes/clustering-guide.md` |
| `migration.stateful` cannot be set while running | Incus requires VM to be stopped | Stop VM first, set config, then start |
| `certificate is valid for 127.0.0.1, ::1, not <IP>` | Cluster cert regenerated on enable/join | Remove and re-add remote with `--accept-certificate` |
| `Config key "zfs.pool_name" is cluster member specific` | Storage pool exists on joining node | Delete pool/network before joining; see clustering guide |
| OC `get-image` fails | No updates in `ready` state yet | Wait ~3 min for OC to download packages; check `provisioning update list` |
| Node doesn't appear in OC server list | ISO not OC-provisioned or token exhausted | Verify ISO from `get-image` with valid token; check `token list` for remaining uses |
| OC web UI returns 403 | Client cert not imported in browser | Import `client.pfx` into browser certificates |
| Bash mangles Proxmox API token with `!` | History expansion | Use single quotes around token value or store in variable |
| OC token seed fields all `{}` | Flat YAML format used | Use structured format: `install: { force_reboot: true }` not `force_reboot: true` at root |
| OC cluster Terraform errors ("already in trust store") | `apply_defaults: true` pre-created resources | Non-fatal -- cluster forms despite errors. Skip `--application-seed-config` or skip `apply_defaults` |
| OC reboot kills OC-managed node (port 8443 closed) | OC agent boot failures: invalid crontab, re-registration conflict, or peers in boot loops (standalone reboot is safe) | Destroy and redeploy OC-managed VM. Guest reboot works fine on standalone IncusOS |
| OC inventory empty after creating workloads | Inventory not real-time | Run `operations-center provisioning cluster resync <name>` |
| OC shows stale server entry after node replacement | Out-of-band cluster change | OC can't detect `incus cluster remove`; stale entries persist |
| OC `server remove` fails ("part of cluster") | Server is in OC cluster | Must remove cluster first or handle via OC cluster management |
| `cluster join` prompts for `tunnel.mesh.interface` | OC adds `meshbr0` network | Add extra `\n` for meshbr0 prompt: `printf '\n\nyes\n\nlocal/incus\nlocal/incus\n'` |
| VMID collision (500/403 on VM create) | Pool-scoped API token can't see VMs outside pool | Use `start_vmid: 400` or higher in config to avoid collisions |