incus-contrib/incusos/TESTING.md

# Testing Guide: incusos-proxmox

A step-by-step guide for deploying IncusOS VMs to Proxmox VE using
`incusos-proxmox`.

## Implementation Status

The three scripts (`incusos-proxmox`, `incusos-seed`, `incusos-iso`) implement
the full lifecycle:

- **Config parsing** -- YAML with PyYAML or built-in fallback parser
- **ISO download** -- fetches latest IncusOS from the CDN by channel
  (stable/testing)
- **Per-VM seed generation** -- calls `incusos-seed --format iso` for each VM
- **Upload to Proxmox** -- SCP (SSH method) or multipart POST (API method)
- **VM creation** -- all IncusOS-required settings enforced (UEFI, TPM,
  VirtIO-SCSI, etc.)
- **Automated install monitoring** -- polls `blockstat.scsi0.wr_bytes` via API
  to detect when disk writes stop (3 stable polls = 15s idle), then stops VM,
  detaches install media (ide2 + ide3), and boots from disk
- **Cleanup** -- safe teardown with managed-VM marker checks

### Cluster Formation (Manual Step)

The `cluster` and `cluster_role` fields are accepted in config for
documentation purposes, but automatic cluster join is not yet implemented.
After deployment, join nodes to the cluster manually:

```bash
# On the init node:
incus cluster add <node-name>
# Copy the join token, then on the joining node's preseed or CLI
```

---

## Authentication

### SSH (Default)

Uses key-based SSH as `root@<host>`. Requires no setup beyond
`ssh-copy-id`.

### API Token (Recommended for Production)

Avoids root SSH entirely. Uses a dedicated Proxmox user with minimal
privileges.

#### Minimum Required Privileges

The role below is already the minimum viable set -- every privilege is
required by at least one operation in the script:

| Privilege | Used For |
|-----------|----------|
| `VM.Allocate` | Create and destroy VMs |
| `VM.Config.Disk` | scsi0, efidisk0, tpmstate0 |
| `VM.Config.CPU` | Set cores, cpu=host |
| `VM.Config.Memory` | Set memory, balloon=0 |
| `VM.Config.Network` | Attach VM to bridge |
| `VM.Config.CDROM` | Attach ISO (ide2) and seed (ide3) |
| `VM.Config.Options` | Set boot order, description, agent flag |
| `VM.Config.HWType` | Set bios=ovmf, machine=q35, scsihw |
| `VM.PowerMgmt` | Start, stop, and reboot VMs |
| `VM.Audit` | Query VM status and config |
| `Datastore.AllocateSpace` | Create disks in storage pool |
| `Datastore.AllocateTemplate` | Upload ISOs to storage |
| `Datastore.Audit` | List and check storage pools |
| `SDN.Use` | Use network bridges |

None of these can be removed without breaking functionality.

#### Setup Commands (Run on Proxmox)

```bash
pveum role add IncusOSDeploy --privs "VM.Allocate VM.Config.Disk \
    VM.Config.CPU VM.Config.Memory VM.Config.Network VM.Config.CDROM \
    VM.Config.Options VM.Config.HWType VM.PowerMgmt VM.Audit \
    Datastore.AllocateSpace Datastore.AllocateTemplate \
    Datastore.Audit SDN.Use"

pveum user add automation@pve
pveum aclmod / -user automation@pve -role IncusOSDeploy
pveum user token add automation@pve deploy --privsep 0
# Save the displayed secret!
```

#### Optional: Scoped ACLs

For tighter access control, scope the ACL to specific paths instead of `/`:

```bash
pveum aclmod /storage/local -user automation@pve -role IncusOSDeploy
pveum aclmod /storage/local-lvm -user automation@pve -role IncusOSDeploy
pveum aclmod /vms -user automation@pve -role IncusOSDeploy
```

For a home lab, `/` (root) is simpler and perfectly acceptable.

---

## Guided Test Scenario

### Prerequisites

**On your workstation** (where you run the script):

```bash
# Verify tools
bash --version           # Need 4+
python3 --version        # For YAML parsing
which jq curl ssh scp    # All needed for SSH method
which genisoimage        # Needed for seed ISO images (used by incusos-proxmox)

# Verify the Incus client has generated a keypair
ls ~/.config/incus/client.crt ~/.config/incus/client.key
# If missing, run:  incus remote list   (triggers auto-generation)
```

**On your Proxmox host**: nothing to install -- just ensure SSH key auth
works.

### Step 1: Verify SSH Connectivity

```bash
ssh root@<YOUR_PROXMOX_IP> pvesh get /version
```

You should see JSON with the Proxmox VE version. If this fails, fix SSH
key auth first (`ssh-copy-id root@<host>`).

### Step 2: Create a Test Config

```bash
cp incusos/examples/proxmox-minimal.yaml my-test.yaml
```

Edit `my-test.yaml` -- change only the host IP:

```yaml
proxmox:
  host: <YOUR_PROXMOX_IP>       # <-- change this

defaults:
  start_vmid: 900                # High range to avoid collisions

vms:
  - name: incus-test
    app: incus
    apply_defaults: true
    cores: 4
    memory: 8192
    disk: 64
```

Using VMID 900+ avoids clashing with existing VMs.

### Step 3: Dry Run

```bash
./incusos/incusos-proxmox --dry-run my-test.yaml
```

**Verify in the output:**

- Config parsed correctly (host, node, storage shown)
- Client certificate detected (`~/.config/incus/client.crt`)
- Storage pools `local-lvm` and `local` found (adjust for your setup)
- Network bridge `vmbr0` found
- VMID 900 allocated for `incus-test`
- ISO version resolved from CDN (e.g., `IncusOS_25.03.iso`)
- `[dry-run]` prefix on all actions -- nothing actually executed
- No errors

### Step 4: Deploy (Single VM)

```bash
./incusos/incusos-proxmox my-test.yaml
```

**Watch the phases:**

1. **Validate** -- connects to Proxmox, checks storage/bridge/tools
2. **Confirm** -- type `y` to proceed
3. **Prepare** -- downloads IncusOS ISO from CDN (~500 MB), generates seed
4. **Upload** -- SCPs ISO + seed to Proxmox
5. **Create** -- runs `qm create 900 ...` with all IncusOS settings
6. **Install** -- starts VM, monitors disk I/O via blockstat, detaches media, boots from disk (~2-3 min)

**Simultaneously on the Proxmox web UI:**

- Open `https://<YOUR_PROXMOX_IP>:8006`
- VM 900 (`incus-test`) should appear
- Open its Console tab
- Watch the IncusOS installer boot from ISO, read the seed, install to
  disk, and reboot
- After reboot you should see the IncusOS login prompt

### Step 5: Verify the Deployed VM

Once the script reports success with an IP address:

```bash
# The cert was injected via seed, so no auth prompt appears
incus remote add incus-test <REPORTED_IP> --accept-certificate

# Test connectivity
incus remote list
incus info incus-test:

# If apply_defaults was true, there should be a ZFS pool and bridge
incus storage list incus-test:
incus network list incus-test:
```

**Note:** IncusOS is immutable and has no QEMU guest agent. IP detection uses
ARP-based lookup (MAC from Proxmox VM config → ARP table). If the IP is wrong,
check the Proxmox console — stale ARP entries can be misleading.

### Step 6: Clean Up the Test VM

```bash
./incusos/incusos-proxmox --cleanup my-test.yaml
```

- Confirms before destroying
- Checks the management marker (`[incusos-lab:managed]` in the VM
  description)
- Stops and destroys with `--purge`

Remove the remote too:

```bash
incus remote remove incus-test
```

### Step 7: (Optional) Test API Method

After single-VM success with SSH, switch to API tokens.

**On Proxmox** (one-time setup via SSH or web shell):

```bash
pveum role add IncusOSDeploy --privs "VM.Allocate VM.Config.Disk \
    VM.Config.CPU VM.Config.Memory VM.Config.Network VM.Config.CDROM \
    VM.Config.Options VM.Config.HWType VM.PowerMgmt VM.Audit \
    Datastore.AllocateSpace Datastore.AllocateTemplate \
    Datastore.Audit SDN.Use"
pveum user add automation@pve
pveum aclmod / -user automation@pve -role IncusOSDeploy
pveum user token add automation@pve deploy --privsep 0
# ^^^ Save the displayed secret!
```

**On your workstation:**

```bash
export PROXMOX_TOKEN_SECRET="<the-secret-from-above>"
```

Update `my-test.yaml`:

```yaml
proxmox:
  host: <YOUR_PROXMOX_IP>
  method: api
  api_token_id: automation@pve!deploy
```

Repeat steps 3-6 using the API method.

### Step 8: (Optional) Full Lab Deployment

Once single-VM tests pass, deploy the full 4-VM lab:

```bash
cp incusos/examples/proxmox-lab.yaml my-lab.yaml
# Edit: change host IP, adjust start_vmid if needed
./incusos/incusos-proxmox --dry-run my-lab.yaml
./incusos/incusos-proxmox my-lab.yaml
```

This deploys 1 Operations Center + 3 Incus nodes. After deployment:

1. Access Operations Center via browser at its reported IP
2. On the init node, run `incus cluster add <node-name>` to get join tokens
3. Join the other two nodes using the tokens

---

## Troubleshooting

| Symptom | Likely Cause | Fix |
|---------|-------------|-----|
| SSH connection refused | Key auth not set up | `ssh-copy-id root@<host>` |
| Storage pool not found | Wrong pool name in config | Check `pvesm status` on Proxmox |
| ISO download fails | CDN unreachable | Download manually, use `--iso path/to/file.iso` |
| VM creation fails | VMID collision | Use higher `start_vmid` or check `qm list` |
| Install times out (10 min) | Slow I/O or boot issue | Check VM console in Proxmox web UI |
| No IP reported | No guest agent (expected) | Check Proxmox console; script uses ARP-based detection |
| Wrong IP reported | Stale ARP entries | Script flushes ARP, but verify via Proxmox console |
| "install media detected" on boot | ide2/ide3 still attached | Script auto-detaches; if manual, delete ide2+ide3 then start |
| "no target device matched" | Explicit `disk-target` in seed | Remove `disk-target`; let IncusOS auto-detect |
| "field server not found" YAML error | Wrong preseed structure | Use `preseed.certificates[]` not `preseed.server.certificates[]` |
| Cert not auto-detected | No Incus keypair | Run `incus remote list` to generate one |
| `genisoimage` not found | genisoimage not installed | Install `genisoimage` package |