incus-contrib/incusos/TESTING.md

# Testing Guide: incusos-proxmox

A step-by-step guide for deploying IncusOS VMs to Proxmox VE using
`incusos-proxmox`.

## Implementation Status

The three scripts (`incusos-proxmox`, `incusos-seed`, `incusos-iso`) implement
the full lifecycle:

- **Config parsing** -- YAML with PyYAML or built-in fallback parser
- **ISO download** -- fetches latest IncusOS from the CDN by channel
  (stable/testing)
- **Per-VM seed generation** -- calls `incusos-seed --format iso` for each VM
- **Upload to Proxmox** -- SCP (SSH method) or multipart POST (API method)
- **VM creation** -- all IncusOS-required settings enforced (UEFI, TPM,
  VirtIO-SCSI, etc.)
- **Automated install monitoring** -- polls `blockstat.scsi0.wr_bytes` via API
  to detect when disk writes stop (3 stable polls = 15s idle), then stops VM,
  detaches install media (ide2 + ide3), and boots from disk
- **Cleanup** -- safe teardown with managed-VM marker checks

### Cluster Formation (Manual Step)

The `cluster` and `cluster_role` fields are accepted in config for
documentation purposes, but automatic cluster join is not yet implemented.
After deployment, join nodes to the cluster manually:

```bash
# On the init node:
incus cluster add <node-name>
# Copy the join token, then on the joining node's preseed or CLI
```

---

## Authentication

### SSH (Default)

Uses key-based SSH as `root@<host>`. Requires no setup beyond
`ssh-copy-id`.

### API Token (Recommended for Production)

Avoids root SSH entirely. Uses a dedicated Proxmox user with minimal
privileges.

#### Minimum Required Privileges

The role below is already the minimum viable set -- every privilege is
required by at least one operation in the script:

| Privilege | Used For |
|-----------|----------|
| `VM.Allocate` | Create and destroy VMs |
| `VM.Config.Disk` | scsi0, efidisk0, tpmstate0 |
| `VM.Config.CPU` | Set cores, cpu=host |
| `VM.Config.Memory` | Set memory, balloon=0 |
| `VM.Config.Network` | Attach VM to bridge |
| `VM.Config.CDROM` | Attach ISO (ide2) and seed (ide3) |
| `VM.Config.Options` | Set boot order, description, agent flag |
| `VM.Config.HWType` | Set bios=ovmf, machine=q35, scsihw |
| `VM.PowerMgmt` | Start, stop, and reboot VMs |
| `VM.Audit` | Query VM status and config |
| `Datastore.AllocateSpace` | Create disks in storage pool |
| `Datastore.AllocateTemplate` | Upload ISOs to storage |
| `Datastore.Audit` | List and check storage pools |
| `SDN.Use` | Use network bridges |

None of these can be removed without breaking functionality.

#### Setup Commands (Run on Proxmox)

```bash
pveum role add IncusOSDeploy --privs "VM.Allocate VM.Config.Disk \
    VM.Config.CPU VM.Config.Memory VM.Config.Network VM.Config.CDROM \
    VM.Config.Options VM.Config.HWType VM.PowerMgmt VM.Audit \
    Datastore.AllocateSpace Datastore.AllocateTemplate \
    Datastore.Audit SDN.Use"

pveum user add automation@pve
pveum aclmod / -user automation@pve -role IncusOSDeploy
pveum user token add automation@pve deploy --privsep 0
# Save the displayed secret!
```

#### Optional: Scoped ACLs

For tighter access control, scope the ACL to specific paths instead of `/`:

```bash
pveum aclmod /storage/local -user automation@pve -role IncusOSDeploy
pveum aclmod /storage/local-lvm -user automation@pve -role IncusOSDeploy
pveum aclmod /vms -user automation@pve -role IncusOSDeploy
```

For a home lab, `/` (root) is simpler and perfectly acceptable.

---

## Guided Test Scenario

### Prerequisites

**On your workstation** (where you run the script):

```bash
# Verify tools
bash --version           # Need 4+
python3 --version        # For YAML parsing
which jq curl ssh scp    # All needed for SSH method
which genisoimage        # Needed for seed ISO images (used by incusos-proxmox)

# Verify the Incus client has generated a keypair
ls ~/.config/incus/client.crt ~/.config/incus/client.key
# If missing, run:  incus remote list   (triggers auto-generation)
```

**On your Proxmox host**: nothing to install -- just ensure SSH key auth
works.

### Step 1: Verify SSH Connectivity

```bash
ssh root@<YOUR_PROXMOX_IP> pvesh get /version
```

You should see JSON with the Proxmox VE version. If this fails, fix SSH
key auth first (`ssh-copy-id root@<host>`).

### Step 2: Set Up Proxmox Connection

```bash
cp incusos/examples/proxmox.yaml.example incusos/proxmox.yaml
```

Edit `incusos/proxmox.yaml` with your Proxmox host settings. This is
auto-discovered by the scripts and avoids repeating credentials in each
lab config.

### Step 3: Create a Test Config

```bash
cp incusos/examples/proxmox-minimal.yaml my-test.yaml
```

Edit `my-test.yaml` -- change only the host IP (or rely on proxmox.yaml):

```yaml
proxmox:
  host: <YOUR_PROXMOX_IP>       # <-- change this

defaults:
  start_vmid: 900                # High range to avoid collisions

vms:
  - name: incus-test
    app: incus
    apply_defaults: true
    cores: 4
    memory: 8192
    disk: 64
```

Using VMID 900+ avoids clashing with existing VMs.

### Step 4: Dry Run

```bash
./incusos/incusos-proxmox --dry-run my-test.yaml
```

**Verify in the output:**

- Config parsed correctly (host, node, storage shown)
- Client certificate detected (`~/.config/incus/client.crt`)
- Storage pools `local-lvm` and `local` found (adjust for your setup)
- Network bridge `vmbr0` found
- VMID 900 allocated for `incus-test`
- ISO version resolved from CDN (e.g., `IncusOS_25.03.iso`)
- `[dry-run]` prefix on all actions -- nothing actually executed
- No errors

### Step 5: Deploy (Single VM)

```bash
./incusos/incusos-proxmox my-test.yaml
```

**Watch the phases:**

1. **Validate** -- connects to Proxmox, checks storage/bridge/tools
2. **Confirm** -- type `y` to proceed
3. **Prepare** -- downloads IncusOS ISO from CDN (~500 MB), generates seed
4. **Upload** -- SCPs ISO + seed to Proxmox
5. **Create** -- runs `qm create 900 ...` with all IncusOS settings
6. **Install** -- starts VM, monitors disk I/O via blockstat, detaches media, boots from disk (~2-3 min)

**Simultaneously on the Proxmox web UI:**

- Open `https://<YOUR_PROXMOX_IP>:8006`
- VM 900 (`incus-test`) should appear
- Open its Console tab
- Watch the IncusOS installer boot from ISO, read the seed, install to
  disk, and reboot
- After reboot you should see the IncusOS login prompt

### Step 6: Check Deployment Status

After the deploy completes, verify with the built-in status command:

```bash
./incusos/incusos-proxmox --status my-test.yaml
```

**Verify in the output:**

- VM `incus-test` exists and is managed
- Status is `running`, install state is `installed`
- IP address detected
- Port 8443 is open
- Incus remote status shown (if configured)

### Step 7: Verify the Deployed VM

Once the script reports success with an IP address:

```bash
# The cert was injected via seed, so no auth prompt appears
incus remote add incus-test <REPORTED_IP> --accept-certificate

# Test connectivity
incus remote list
incus info incus-test:

# If apply_defaults was true, there should be a ZFS pool and bridge
incus storage list incus-test:
incus network list incus-test:
```

**Note:** IncusOS is immutable and has no QEMU guest agent. IP detection uses
ARP-based lookup (MAC from Proxmox VM config → ARP table). If the IP is wrong,
check the Proxmox console — stale ARP entries can be misleading.

### Step 8: Test Re-run Reconcile

Re-run the deploy to verify the reconcile menu:

```bash
./incusos/incusos-proxmox my-test.yaml
```

**Verify:** The script detects the existing VM and shows the interactive menu.
Choose option 1 to run status checks without modifying anything.

Also test with `--yes` to verify it defaults to safe behavior:

```bash
./incusos/incusos-proxmox --yes my-test.yaml
```

**Verify:** Runs post-deployment checks, does NOT destroy.

### Step 9: Clean Up the Test VM

```bash
./incusos/incusos-proxmox --cleanup my-test.yaml
```

- Confirms before destroying
- Checks the management marker (`[incusos-lab:managed]` in the VM
  description)
- Stops and destroys with `--purge`

Remove the remote too:

```bash
incus remote remove incus-test
```

### Step 10: (Optional) Test API Method

After single-VM success with SSH, switch to API tokens.

**On Proxmox** (one-time setup via SSH or web shell):

```bash
pveum role add IncusOSDeploy --privs "VM.Allocate VM.Config.Disk \
    VM.Config.CPU VM.Config.Memory VM.Config.Network VM.Config.CDROM \
    VM.Config.Options VM.Config.HWType VM.PowerMgmt VM.Audit \
    Datastore.AllocateSpace Datastore.AllocateTemplate \
    Datastore.Audit SDN.Use"
pveum user add automation@pve
pveum aclmod / -user automation@pve -role IncusOSDeploy
pveum user token add automation@pve deploy --privsep 0
# ^^^ Save the displayed secret!
```

**On your workstation:**

```bash
export PROXMOX_TOKEN_SECRET="<the-secret-from-above>"
```

Update `my-test.yaml`:

```yaml
proxmox:
  host: <YOUR_PROXMOX_IP>
  method: api
  api_token_id: automation@pve!deploy
```

Repeat steps 4-7 using the API method.

### Step 11: (Optional) Test with Resource Pool

For API method with pool isolation:

**On Proxmox:**

```bash
pveum pool add IncusLab --comment "IncusOS Lab VMs"
pveum pool modify IncusLab --storage local-lvm,local
pveum aclmod /pool/IncusLab -user automation@pve -role IncusOSDeploy
```

Add to your config:

```yaml
proxmox:
  pool: IncusLab
```

Deploy and verify:

```bash
./incusos/incusos-proxmox --dry-run my-test.yaml   # Should show pool in plan
./incusos/incusos-proxmox my-test.yaml              # VM created in pool
./incusos/incusos-proxmox --status my-test.yaml     # Status scoped to pool
```

### Step 12: (Optional) Full Lab Deployment

Once single-VM tests pass, deploy the full 4-VM lab:

```bash
cp incusos/examples/proxmox-lab.yaml my-lab.yaml
# Edit: change host IP, adjust start_vmid if needed
./incusos/incusos-proxmox --dry-run my-lab.yaml
./incusos/incusos-proxmox my-lab.yaml
```

This deploys 1 Operations Center + 3 Incus nodes. After deployment:

1. Access Operations Center via browser at its reported IP
2. On the init node, run `incus cluster add <node-name>` to get join tokens
3. Join the other two nodes using the tokens

---

## Lab Validation with lab-test

The `lab-test` script automates the end-to-end validation of deployed IncusOS
VMs: single-node workloads, cluster formation, migration, and evacuation.

### Quick Start: Cluster Testing

```bash
# 1. Check environment
./incusos/incusos-proxmox --doctor

# 2. Deploy 3 nodes
./incusos/incusos-proxmox --yes examples/lab-cluster.yaml

# 3. Add incus remotes for each node (use IPs from --status)
./incusos/incusos-proxmox --status examples/lab-cluster.yaml
incus remote add incus-lab-01 <IP1> --accept-certificate
incus remote add incus-lab-02 <IP2> --accept-certificate
incus remote add incus-lab-03 <IP3> --accept-certificate

# 4. Run full lab validation
./incusos/lab-test --yes examples/lab-cluster.yaml
```

### Phase-by-Phase Testing

```bash
# Single-node workloads only (container + VM launch/exec)
./incusos/lab-test --phase single --skip-deploy examples/lab-cluster.yaml

# Cluster formation only
./incusos/lab-test --phase cluster --skip-deploy examples/lab-cluster.yaml

# Cluster workloads (scheduler placement + targeted placement)
./incusos/lab-test --phase workload --skip-deploy examples/lab-cluster.yaml

# Migration and evacuation tests
./incusos/lab-test --phase migrate --skip-deploy examples/lab-cluster.yaml

# Extended phases (require a working cluster):
./incusos/lab-test --phase storage --skip-deploy examples/lab-cluster.yaml
./incusos/lab-test --phase network --skip-deploy examples/lab-cluster.yaml
./incusos/lab-test --phase projects --skip-deploy examples/lab-cluster.yaml
./incusos/lab-test --phase backup --skip-deploy examples/lab-cluster.yaml
./incusos/lab-test --phase limits --skip-deploy examples/lab-cluster.yaml
./incusos/lab-test --phase security --skip-deploy examples/lab-cluster.yaml
./incusos/lab-test --phase resilience --skip-deploy examples/lab-cluster.yaml
```

### Extended Test Phases

| Phase | Tests | Requires |
|-------|-------|----------|
| `storage` | Pool listing, volume create/delete, volume snapshots, pool usage | Single node |
| `network` | Network listing, bridge create/delete, ACLs, forwards | Single node |
| `projects` | Project create with limits, custom profiles, profile inheritance | Single node |
| `backup` | Instance snapshots (create/restore/delete), export/import round-trip | Single node |
| `limits` | CPU limits, memory limits, disk I/O limits | Single node |
| `security` | Unprivileged vs privileged containers, nesting, user namespace isolation | Single node |
| `resilience` | Cluster health, database leader, member state, warnings | 3-node cluster |

### Manual Cluster Formation (Alternative)

If you prefer to form the cluster manually instead of using `lab-test`,
follow the steps below. See also `notes/clustering-guide.md` for the full
reference guide with explanations of what happens under the hood.

#### Prerequisites

```bash
# Check client version (need 6.20+ for remote cluster join)
incus version

# Verify all three remotes are working
incus info incus-lab-01: | head -5
incus info incus-lab-02: | head -5
incus info incus-lab-03: | head -5
```

#### Step 1: Set specific IPs on all nodes

IncusOS defaults to `core.https_address: :8443` (wildcard). Clustering
requires a specific routable IP on each node.

```bash
# Discover each node's routable IP
for node in incus-lab-01 incus-lab-02 incus-lab-03; do
  echo "=== $node ==="
  incus query "$node:/1.0" | python3 -c "import sys,json; d=json.load(sys.stdin); \
    [print(a) for a in d['environment']['addresses'] \
     if not a.startswith('10.') and not a.startswith('fd42:') and not a.startswith('[')]"
done

# Set the specific IP on each node (replace with actual IPs)
incus config set incus-lab-01: core.https_address <IP1>:8443
incus config set incus-lab-02: core.https_address <IP2>:8443
incus config set incus-lab-03: core.https_address <IP3>:8443

# Verify connectivity still works after each change
incus info incus-lab-01: | head -3
incus info incus-lab-02: | head -3
incus info incus-lab-03: | head -3
```

#### Step 2: Enable clustering on the init node

```bash
incus cluster enable incus-lab-01:incus-lab-01
```

> **Important**: This regenerates the server's TLS certificate. Your remote
> will break with a certificate error. Fix it:

```bash
incus remote switch local          # if incus-lab-01 is the current default
incus remote remove incus-lab-01
incus remote add incus-lab-01 https://<IP1>:8443 --accept-certificate
```

Verify:

```bash
incus cluster list incus-lab-01:
# Should show one member (incus-lab-01), status ONLINE, role database-leader
```

#### Step 3: Prepare joining nodes

Nodes with `apply_defaults: true` have a pre-existing `local` storage pool
and `incusbr0` network. These conflict with the cluster join process (the
join wizard tries to set member-specific ZFS config keys at cluster level).
Delete them before joining:

```bash
# For each joining node (incus-lab-02, incus-lab-03):
NODE=incus-lab-02   # then repeat with incus-lab-03

# Remove server-level references to the pool
incus config unset "$NODE": storage.backups_volume
incus config unset "$NODE": storage.images_volume

# Delete the backup and image volumes
incus storage volume delete "$NODE":local backups
incus storage volume delete "$NODE":local images

# Clear default profile references (otherwise pool is "in use")
incus profile device remove "$NODE":default root
incus profile device remove "$NODE":default eth0

# Delete pool and network
incus storage delete "$NODE":local
incus network delete "$NODE":incusbr0
```

#### Step 4: Join nodes to the cluster

**Interactive** (one node at a time):

```bash
# Generate join token
incus cluster add incus-lab-01:incus-lab-02

# Join (answer the interactive prompts)
incus cluster join incus-lab-01: incus-lab-02:
# Prompt 1 - IP address: press Enter (accept default)
# Prompt 2 - Member name: press Enter (accept default)
# Prompt 3 - "All existing data is lost": type "yes"
# Prompt 4 - source for storage pool "local": type "local/incus"
# Prompt 5 - zfs.pool_name for storage pool "local": type "local/incus"
```

**Automated** (non-interactive):

```bash
# Generate token + join in one go
incus cluster add incus-lab-01:incus-lab-02
printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join incus-lab-01: incus-lab-02:
```

After each join, fix the remote (same certificate issue as the init node):

```bash
incus remote remove incus-lab-02
incus remote add incus-lab-02 https://<IP2>:8443 --accept-certificate
```

Repeat for incus-lab-03.

#### Step 5: Verify the cluster

```bash
# All 3 members should be ONLINE
incus cluster list incus-lab-01:

# Storage pool should show on all members
incus storage show incus-lab-01:local
incus storage show incus-lab-01:local --target incus-lab-01
incus storage show incus-lab-01:local --target incus-lab-02
incus storage show incus-lab-01:local --target incus-lab-03

# All remotes should work
incus info incus-lab-01: | head -3
incus info incus-lab-02: | head -3
incus info incus-lab-03: | head -3
```

**Key points:**
- `apply_defaults: true` on all nodes ensures matching storage pool and
  network bridge names (required for migration) -- but the pools must be
  deleted on joining nodes before the join
- After joining, manage the cluster through the init node's remote
- The ZFS pool source is `local/incus` on IncusOS (auto-created by
  `apply_defaults`)
- Each join generates a new token (tokens are single-use and time-limited)
- VM migration requires matching storage pool names across nodes

### Migration Testing (After Cluster Formation)

Once the cluster is formed, validate container and VM migration:

#### Container migration (stop/move/start)

```bash
# Launch on node 2
incus launch images:debian/12 incus-lab-01:test-container --target incus-lab-02
incus exec incus-lab-01:test-container -- bash -c 'echo "hello" > /root/test.txt'

# Migrate to node 3
incus stop incus-lab-01:test-container
incus move incus-lab-01:test-container --target incus-lab-03
incus start incus-lab-01:test-container

# Verify: data survives, processes don't
incus exec incus-lab-01:test-container -- cat /root/test.txt   # "hello"
incus list incus-lab-01: --columns nLs                         # node 3
```

#### VM live migration

```bash
# Launch and configure for migration
incus launch images:debian/12 incus-lab-01:test-vm --vm --target incus-lab-02
# Wait for VM agent...
incus stop incus-lab-01:test-vm
incus config set incus-lab-01:test-vm migration.stateful=true
incus config set incus-lab-01:test-vm limits.cpu=0-1
incus config device add incus-lab-01:test-vm root disk path=/ pool=local size.state=2GiB
incus start incus-lab-01:test-vm

# Set up heartbeat for state verification
incus exec incus-lab-01:test-vm -- mkdir -p /tmp/migration-test
incus exec incus-lab-01:test-vm -- bash -c \
  'cat > /tmp/migration-test/heartbeat.sh << "EOF"
#!/bin/bash
i=0; while true; do echo $i > /tmp/migration-test/heartbeat; i=$((i+1)); sleep 1; done
EOF
chmod +x /tmp/migration-test/heartbeat.sh'
incus exec incus-lab-01:test-vm -- \
  systemd-run --unit=migration-heartbeat /tmp/migration-test/heartbeat.sh

# Live-migrate (state preserved)
sleep 3
incus exec incus-lab-01:test-vm -- cat /tmp/migration-test/heartbeat   # note value
incus move incus-lab-01:test-vm --target incus-lab-03
sleep 2   # agent needs ~1-2s to reconnect
incus exec incus-lab-01:test-vm -- cat /tmp/migration-test/heartbeat   # should be higher
incus exec incus-lab-01:test-vm -- systemctl is-active migration-heartbeat  # "active"
```

#### Cluster evacuation

```bash
# Move both workloads to node 2, then evacuate
incus cluster evacuate incus-lab-01:incus-lab-02 --force
incus cluster list incus-lab-01:    # node 2 = EVACUATED
incus list incus-lab-01: --columns nLs  # workloads on other nodes

# Restore
incus cluster restore incus-lab-01:incus-lab-02 --force
incus cluster list incus-lab-01:    # node 2 = ONLINE
incus list incus-lab-01: --columns nLs  # workloads back on node 2
```

See `notes/clustering-guide.md` for the full automated migration test script
and detailed test results.

### Cleanup

```bash
# Remove test instances only (keeps VMs and cluster)
./incusos/lab-test --cleanup examples/lab-cluster.yaml

# Destroy VMs only
./incusos/incusos-proxmox --cleanup examples/lab-cluster.yaml

# Full cleanup: VMs + deployment ISO + seeds + remotes + cache
./incusos/incusos-proxmox --cleanup --deep --yes examples/lab-cluster.yaml

# Pool-wide: destroy ALL managed VMs (no config file needed)
./incusos/incusos-proxmox --cleanup-all --yes
./incusos/incusos-proxmox --cleanup-all --deep --yes   # nuke everything
```

---

## Operations Center Lab (Tested)

### Overview

Operations Center (OC) manages IncusOS nodes through a provisioning pipeline:
tokens → provisioned ISOs → self-registration → cluster formation.

**Hybrid approach** (tested, recommended): use `incusos-proxmox --iso` to
deploy nodes from an OC-provisioned ISO. Combines OC auto-registration with
`incusos-proxmox` deployment automation (VM creation, SEED_DATA, install
monitoring, media cleanup, IP detection).

### Prerequisites

```bash
# 1. Install OC CLI
wget https://github.com/FuturFusion/operations-center/releases/download/v0.2.2/operations-center_linux_x86_64
chmod +x operations-center_linux_x86_64
sudo mv operations-center_linux_x86_64 /usr/local/bin/operations-center

# 2. Set up OC client certs
mkdir -p ~/.config/operations-center
cp ~/.config/incus/client.crt ~/.config/operations-center/
cp ~/.config/incus/client.key ~/.config/operations-center/

# 3. Browser cert (for web UI)
openssl pkcs12 -export \
    -inkey ~/.config/incus/client.key \
    -in ~/.config/incus/client.crt \
    -out ~/.config/incus/client.pfx

# 4. Verify
./incusos/incusos-proxmox --doctor
```

### Step 1: Deploy OC Server

```bash
./incusos/incusos-proxmox --yes incusos/examples/lab-oc-deploy.yaml
./incusos/incusos-proxmox --status incusos/examples/lab-oc-deploy.yaml
# Note OC_IP from output
```

### Step 2: Connect OC CLI

```bash
printf "yes\n" | operations-center remote add oc-lab https://<OC_IP>:8443 --auth-type tls
operations-center remote switch oc-lab
```

### Step 3: Wait for Updates (~3 min)

```bash
operations-center provisioning update list
# Wait until at least one shows "ready"
```

### Step 4: Create Token + Seed

```bash
# Create token
operations-center provisioning token add --description "Lab cluster" --uses 5 --lifetime 168h
operations-center provisioning token list   # Note UUID

# Create structured pre-seed (MUST use section-level keys)
cat > /tmp/oc-preseed.yaml << 'EOF'
install:
  version: "1"
  force_install: true
  force_reboot: true
EOF

# Attach as named seed
operations-center provisioning token seed add <UUID> proxmox-preseed \
    /tmp/oc-preseed.yaml --description "Force reboot for Proxmox VMs"
```

### Step 5: Download OC-Provisioned ISO

```bash
operations-center provisioning token seed get-image <UUID> proxmox-preseed \
    /tmp/IncusOS-oc.iso --type iso --architecture x86_64
ls -lh /tmp/IncusOS-oc.iso   # ~3.4 GB
```

### Step 6: Deploy Nodes (Hybrid)

```bash
./incusos/incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes \
    incusos/examples/lab-oc-nodes.yaml
./incusos/incusos-proxmox --status incusos/examples/lab-oc-nodes.yaml
```

`incusos-proxmox` creates VMs, uploads the OC ISO to ide2, generates per-node
SEED_DATA on ide3 (hostname + force_reboot + app selection + cert), monitors
install, removes media, boots from disk, and sets up incus remotes.

### Step 7: Verify Auto-Registration

```bash
# Nodes register within ~30s of first boot
operations-center provisioning server list
# Should show 3 nodes + OC itself, all "ready"
```

### Step 8: Form Cluster

```bash
# Prepare application seed (client cert for incus CLI access)
cat > /tmp/oc-app-config.yaml << EOF
certificates:
  - type: client
    name: lab-client
    certificate: |-
$(sed 's/^/      /' ~/.config/incus/client.crt)
EOF

# Form cluster
operations-center provisioning cluster add oc-cluster \
    https://<NODE_01_IP>:8443 \
    --server-names oc-node-01,oc-node-02,oc-node-03 \
    --server-type incus \
    --application-seed-config /tmp/oc-app-config.yaml
```

**Note**: with `apply_defaults: true` in SEED_DATA, Terraform post-config
errors are expected ("already in trust store", "not in pending state"). The
cluster forms successfully despite these.

### Step 9: Verify

```bash
# OC
operations-center provisioning cluster list
operations-center provisioning cluster show oc-cluster

# Incus (re-add remotes after cert change from clustering)
incus remote remove oc-node-01 && incus remote add oc-node-01 https://<IP>:8443 --accept-certificate
incus cluster list oc-node-01:

# Inventory (requires resync)
operations-center provisioning cluster resync oc-cluster
operations-center inventory instance list
operations-center inventory storage-pool list
operations-center inventory network list
```

### Step 10: Test Features

```bash
# Workload sync: create via incus, resync, verify in OC
incus launch images:debian/12 oc-node-01:test-ct --target oc-node-01
operations-center provisioning cluster resync oc-cluster
operations-center inventory instance list

# Live migration visibility
incus move oc-node-01:test-vm --target oc-node-03
operations-center provisioning cluster resync oc-cluster
operations-center inventory instance list   # shows new location

# Evacuation
incus cluster evacuate oc-node-01:oc-node-01 --force
operations-center provisioning cluster resync oc-cluster
operations-center inventory instance list   # shows relocated workloads
incus cluster restore oc-node-01:oc-node-01 --force
```

**WARNING**: do NOT use `operations-center provisioning server system reboot`.
It sends a guest-level reboot that likely corrupts the dqlite/Incus database
through unclean shutdown on Proxmox VMs. The node becomes pingable (OS boots)
but Incus daemon never starts (port 8443 closed). This is an open research
item. The node must be destroyed/redeployed.

### Cleanup

```bash
# Test workloads
incus delete oc-node-01:test-ct --force 2>/dev/null
incus delete oc-node-01:test-vm --force 2>/dev/null

# Nodes
./incusos/incusos-proxmox --cleanup --deep --yes incusos/examples/lab-oc-nodes.yaml

# OC server
./incusos/incusos-proxmox --cleanup --deep --yes incusos/examples/lab-oc-deploy.yaml

# Remotes
operations-center remote remove oc-lab 2>/dev/null
```

**Note:** Operations Center is under active development (v0.2.2+). See
`notes/operations-center-guide.md` for the full tested reference.

---

## Troubleshooting

| Symptom | Likely Cause | Fix |
|---------|-------------|-----|
| SSH connection refused | Key auth not set up | `ssh-copy-id root@<host>` |
| Storage pool not found | Wrong pool name in config | Check `pvesm status` on Proxmox |
| ISO download fails | CDN unreachable | Download manually, use `--iso path/to/file.iso` |
| VM creation fails | VMID collision | Use higher `start_vmid` or check `qm list` |
| Install times out (10 min) | Slow I/O or boot issue | Check VM console in Proxmox web UI |
| No IP reported | No guest agent (expected) | Check Proxmox console; script uses ARP-based detection |
| Wrong IP reported | Stale ARP entries | Script flushes ARP, but verify via Proxmox console |
| "install media detected" on boot | ide2/ide3 still attached | Script auto-detaches; if manual, delete ide2+ide3 then start |
| "no target device matched" | Explicit `disk-target` in seed | Remove `disk-target`; let IncusOS auto-detect |
| "field server not found" YAML error | Wrong preseed structure | Use `preseed.certificates[]` not `preseed.server.certificates[]` |
| Cert not auto-detected | No Incus keypair | Run `incus remote list` to generate one |
| `genisoimage` not found | genisoimage not installed | Install `genisoimage` package |
| Pool not found | Resource pool not created | Run `pveum pool add <name>` on Proxmox |
| Re-run destroys VMs | Used option 3 in reconcile | Use option 1 (checks) or option 2 (continue) instead |
| `--status` shows "not found" | VM was cleaned up | Deploy first, then check status |
| VM live migration fails: "Missing section footer for ICH9LPC" | `limits.cpu` not set or set as integer | Use range syntax: `limits.cpu=0-1` (not `2`); see `notes/clustering-guide.md` |
| `migration.stateful` cannot be set while running | Incus requires VM to be stopped | Stop VM first, set config, then start |
| `certificate is valid for 127.0.0.1, ::1, not <IP>` | Cluster cert regenerated on enable/join | Remove and re-add remote with `--accept-certificate` |
| `Config key "zfs.pool_name" is cluster member specific` | Storage pool exists on joining node | Delete pool/network before joining; see clustering guide |
| OC `get-image` fails | No updates in `ready` state yet | Wait ~3 min for OC to download packages; check `provisioning update list` |
| Node doesn't appear in OC server list | ISO not OC-provisioned or token exhausted | Verify ISO from `get-image` with valid token; check `token list` for remaining uses |
| OC web UI returns 403 | Client cert not imported in browser | Import `client.pfx` into browser certificates |
| Bash mangles Proxmox API token with `!` | History expansion | Use single quotes around token value or store in variable |
| OC token seed fields all `{}` | Flat YAML format used | Use structured format: `install: { force_reboot: true }` not `force_reboot: true` at root |
| OC cluster Terraform errors ("already in trust store") | `apply_defaults: true` pre-created resources | Non-fatal -- cluster forms despite errors. Skip `--application-seed-config` or skip `apply_defaults` |
| OC reboot kills IncusOS node (port 8443 closed forever) | Likely dqlite/Incus database corruption from unclean shutdown (open research item) | Destroy and redeploy VM. Use Proxmox stop/start instead of OC reboot |
| OC inventory empty after creating workloads | Inventory not real-time | Run `operations-center provisioning cluster resync <name>` |
| OC shows stale server entry after node replacement | Out-of-band cluster change | OC can't detect `incus cluster remove`; stale entries persist |
| OC `server remove` fails ("part of cluster") | Server is in OC cluster | Must remove cluster first or handle via OC cluster management |
| `cluster join` prompts for `tunnel.mesh.interface` | OC adds `meshbr0` network | Add extra `\n` for meshbr0 prompt: `printf '\n\nyes\n\nlocal/incus\nlocal/incus\n'` |
| VMID collision (500/403 on VM create) | Pool-scoped API token can't see VMs outside pool | Use `start_vmid: 400` or higher in config to avoid collisions |