510 lines
15 KiB
Markdown
510 lines
15 KiB
Markdown
# Testing Guide: incusos-proxmox
|
|
|
|
A step-by-step guide for deploying IncusOS VMs to Proxmox VE using
|
|
`incusos-proxmox`.
|
|
|
|
## Implementation Status
|
|
|
|
The three scripts (`incusos-proxmox`, `incusos-seed`, `incusos-iso`) implement
|
|
the full lifecycle:
|
|
|
|
- **Config parsing** -- YAML with PyYAML or built-in fallback parser
|
|
- **ISO download** -- fetches latest IncusOS from the CDN by channel
|
|
(stable/testing)
|
|
- **Per-VM seed generation** -- calls `incusos-seed --format iso` for each VM
|
|
- **Upload to Proxmox** -- SCP (SSH method) or multipart POST (API method)
|
|
- **VM creation** -- all IncusOS-required settings enforced (UEFI, TPM,
|
|
VirtIO-SCSI, etc.)
|
|
- **Automated install monitoring** -- polls `blockstat.scsi0.wr_bytes` via API
|
|
to detect when disk writes stop (3 stable polls = 15s idle), then stops VM,
|
|
detaches install media (ide2 + ide3), and boots from disk
|
|
- **Cleanup** -- safe teardown with managed-VM marker checks
|
|
|
|
### Cluster Formation (Manual Step)
|
|
|
|
The `cluster` and `cluster_role` fields are accepted in config for
|
|
documentation purposes, but automatic cluster join is not yet implemented.
|
|
After deployment, join nodes to the cluster manually:
|
|
|
|
```bash
|
|
# On the init node:
|
|
incus cluster add <node-name>
|
|
# Copy the join token, then on the joining node's preseed or CLI
|
|
```
|
|
|
|
---
|
|
|
|
## Authentication
|
|
|
|
### SSH (Default)
|
|
|
|
Uses key-based SSH as `root@<host>`. Requires no setup beyond
|
|
`ssh-copy-id`.
|
|
|
|
### API Token (Recommended for Production)
|
|
|
|
Avoids root SSH entirely. Uses a dedicated Proxmox user with minimal
|
|
privileges.
|
|
|
|
#### Minimum Required Privileges
|
|
|
|
The role below is already the minimum viable set -- every privilege is
|
|
required by at least one operation in the script:
|
|
|
|
| Privilege | Used For |
|
|
|-----------|----------|
|
|
| `VM.Allocate` | Create and destroy VMs |
|
|
| `VM.Config.Disk` | scsi0, efidisk0, tpmstate0 |
|
|
| `VM.Config.CPU` | Set cores, cpu=host |
|
|
| `VM.Config.Memory` | Set memory, balloon=0 |
|
|
| `VM.Config.Network` | Attach VM to bridge |
|
|
| `VM.Config.CDROM` | Attach ISO (ide2) and seed (ide3) |
|
|
| `VM.Config.Options` | Set boot order, description, agent flag |
|
|
| `VM.Config.HWType` | Set bios=ovmf, machine=q35, scsihw |
|
|
| `VM.PowerMgmt` | Start, stop, and reboot VMs |
|
|
| `VM.Audit` | Query VM status and config |
|
|
| `Datastore.AllocateSpace` | Create disks in storage pool |
|
|
| `Datastore.AllocateTemplate` | Upload ISOs to storage |
|
|
| `Datastore.Audit` | List and check storage pools |
|
|
| `SDN.Use` | Use network bridges |
|
|
|
|
None of these can be removed without breaking functionality.
|
|
|
|
#### Setup Commands (Run on Proxmox)
|
|
|
|
```bash
|
|
pveum role add IncusOSDeploy --privs "VM.Allocate VM.Config.Disk \
|
|
VM.Config.CPU VM.Config.Memory VM.Config.Network VM.Config.CDROM \
|
|
VM.Config.Options VM.Config.HWType VM.PowerMgmt VM.Audit \
|
|
Datastore.AllocateSpace Datastore.AllocateTemplate \
|
|
Datastore.Audit SDN.Use"
|
|
|
|
pveum user add automation@pve
|
|
pveum aclmod / -user automation@pve -role IncusOSDeploy
|
|
pveum user token add automation@pve deploy --privsep 0
|
|
# Save the displayed secret!
|
|
```
|
|
|
|
#### Optional: Scoped ACLs
|
|
|
|
For tighter access control, scope the ACL to specific paths instead of `/`:
|
|
|
|
```bash
|
|
pveum aclmod /storage/local -user automation@pve -role IncusOSDeploy
|
|
pveum aclmod /storage/local-lvm -user automation@pve -role IncusOSDeploy
|
|
pveum aclmod /vms -user automation@pve -role IncusOSDeploy
|
|
```
|
|
|
|
For a home lab, `/` (root) is simpler and perfectly acceptable.
|
|
|
|
---
|
|
|
|
## Guided Test Scenario
|
|
|
|
### Prerequisites
|
|
|
|
**On your workstation** (where you run the script):
|
|
|
|
```bash
|
|
# Verify tools
|
|
bash --version # Need 4+
|
|
python3 --version # For YAML parsing
|
|
which jq curl ssh scp # All needed for SSH method
|
|
which genisoimage # Needed for seed ISO images (used by incusos-proxmox)
|
|
|
|
# Verify the Incus client has generated a keypair
|
|
ls ~/.config/incus/client.crt ~/.config/incus/client.key
|
|
# If missing, run: incus remote list (triggers auto-generation)
|
|
```
|
|
|
|
**On your Proxmox host**: nothing to install -- just ensure SSH key auth
|
|
works.
|
|
|
|
### Step 1: Verify SSH Connectivity
|
|
|
|
```bash
|
|
ssh root@<YOUR_PROXMOX_IP> pvesh get /version
|
|
```
|
|
|
|
You should see JSON with the Proxmox VE version. If this fails, fix SSH
|
|
key auth first (`ssh-copy-id root@<host>`).
|
|
|
|
### Step 2: Create a Test Config
|
|
|
|
```bash
|
|
cp incusos/examples/proxmox-minimal.yaml my-test.yaml
|
|
```
|
|
|
|
Edit `my-test.yaml` -- change only the host IP:
|
|
|
|
```yaml
|
|
proxmox:
|
|
host: <YOUR_PROXMOX_IP> # <-- change this
|
|
|
|
defaults:
|
|
start_vmid: 900 # High range to avoid collisions
|
|
|
|
vms:
|
|
- name: incus-test
|
|
app: incus
|
|
apply_defaults: true
|
|
cores: 4
|
|
memory: 8192
|
|
disk: 64
|
|
```
|
|
|
|
Using VMID 900+ avoids clashing with existing VMs.
|
|
|
|
### Step 3: Dry Run
|
|
|
|
```bash
|
|
./incusos/incusos-proxmox --dry-run my-test.yaml
|
|
```
|
|
|
|
**Verify in the output:**
|
|
|
|
- Config parsed correctly (host, node, storage shown)
|
|
- Client certificate detected (`~/.config/incus/client.crt`)
|
|
- Storage pools `local-lvm` and `local` found (adjust for your setup)
|
|
- Network bridge `vmbr0` found
|
|
- VMID 900 allocated for `incus-test`
|
|
- ISO version resolved from CDN (e.g., `IncusOS_25.03.iso`)
|
|
- `[dry-run]` prefix on all actions -- nothing actually executed
|
|
- No errors
|
|
|
|
### Step 4: Deploy (Single VM)
|
|
|
|
```bash
|
|
./incusos/incusos-proxmox my-test.yaml
|
|
```
|
|
|
|
**Watch the phases:**
|
|
|
|
1. **Validate** -- connects to Proxmox, checks storage/bridge/tools
|
|
2. **Confirm** -- type `y` to proceed
|
|
3. **Prepare** -- downloads IncusOS ISO from CDN (~500 MB), generates seed
|
|
4. **Upload** -- SCPs ISO + seed to Proxmox
|
|
5. **Create** -- runs `qm create 900 ...` with all IncusOS settings
|
|
6. **Install** -- starts VM, monitors disk I/O via blockstat, detaches media, boots from disk (~2-3 min)
|
|
|
|
**Simultaneously on the Proxmox web UI:**
|
|
|
|
- Open `https://<YOUR_PROXMOX_IP>:8006`
|
|
- VM 900 (`incus-test`) should appear
|
|
- Open its Console tab
|
|
- Watch the IncusOS installer boot from ISO, read the seed, install to
|
|
disk, and reboot
|
|
- After reboot you should see the IncusOS login prompt
|
|
|
|
### Step 5: Check Deployment Status
|
|
|
|
After the deploy completes, verify with the built-in status command:
|
|
|
|
```bash
|
|
./incusos/incusos-proxmox --status my-test.yaml
|
|
```
|
|
|
|
**Verify in the output:**
|
|
|
|
- VM `incus-test` exists and is managed
|
|
- Status is `running`, install state is `installed`
|
|
- IP address detected
|
|
- Port 8443 is open
|
|
- Incus remote status shown (if configured)
|
|
|
|
### Step 6: Verify the Deployed VM
|
|
|
|
Once the script reports success with an IP address:
|
|
|
|
```bash
|
|
# The cert was injected via seed, so no auth prompt appears
|
|
incus remote add incus-test <REPORTED_IP> --accept-certificate
|
|
|
|
# Test connectivity
|
|
incus remote list
|
|
incus info incus-test:
|
|
|
|
# If apply_defaults was true, there should be a ZFS pool and bridge
|
|
incus storage list incus-test:
|
|
incus network list incus-test:
|
|
```
|
|
|
|
**Note:** IncusOS is immutable and has no QEMU guest agent. IP detection uses
|
|
ARP-based lookup (MAC from Proxmox VM config → ARP table). If the IP is wrong,
|
|
check the Proxmox console — stale ARP entries can be misleading.
|
|
|
|
### Step 7: Test Re-run Reconcile
|
|
|
|
Re-run the deploy to verify the reconcile menu:
|
|
|
|
```bash
|
|
./incusos/incusos-proxmox my-test.yaml
|
|
```
|
|
|
|
**Verify:** The script detects the existing VM and shows the interactive menu.
|
|
Choose option 1 to run status checks without modifying anything.
|
|
|
|
Also test with `--yes` to verify it defaults to safe behavior:
|
|
|
|
```bash
|
|
./incusos/incusos-proxmox --yes my-test.yaml
|
|
```
|
|
|
|
**Verify:** Runs post-deployment checks, does NOT destroy.
|
|
|
|
### Step 8: Clean Up the Test VM
|
|
|
|
```bash
|
|
./incusos/incusos-proxmox --cleanup my-test.yaml
|
|
```
|
|
|
|
- Confirms before destroying
|
|
- Checks the management marker (`[incusos-lab:managed]` in the VM
|
|
description)
|
|
- Stops and destroys with `--purge`
|
|
|
|
Remove the remote too:
|
|
|
|
```bash
|
|
incus remote remove incus-test
|
|
```
|
|
|
|
### Step 9: (Optional) Test API Method
|
|
|
|
After single-VM success with SSH, switch to API tokens.
|
|
|
|
**On Proxmox** (one-time setup via SSH or web shell):
|
|
|
|
```bash
|
|
pveum role add IncusOSDeploy --privs "VM.Allocate VM.Config.Disk \
|
|
VM.Config.CPU VM.Config.Memory VM.Config.Network VM.Config.CDROM \
|
|
VM.Config.Options VM.Config.HWType VM.PowerMgmt VM.Audit \
|
|
Datastore.AllocateSpace Datastore.AllocateTemplate \
|
|
Datastore.Audit SDN.Use"
|
|
pveum user add automation@pve
|
|
pveum aclmod / -user automation@pve -role IncusOSDeploy
|
|
pveum user token add automation@pve deploy --privsep 0
|
|
# ^^^ Save the displayed secret!
|
|
```
|
|
|
|
**On your workstation:**
|
|
|
|
```bash
|
|
export PROXMOX_TOKEN_SECRET="<the-secret-from-above>"
|
|
```
|
|
|
|
Update `my-test.yaml`:
|
|
|
|
```yaml
|
|
proxmox:
|
|
host: <YOUR_PROXMOX_IP>
|
|
method: api
|
|
api_token_id: automation@pve!deploy
|
|
```
|
|
|
|
Repeat steps 3-6 using the API method.
|
|
|
|
### Step 10: (Optional) Test with Resource Pool
|
|
|
|
For API method with pool isolation:
|
|
|
|
**On Proxmox:**
|
|
|
|
```bash
|
|
pveum pool add IncusLab --comment "IncusOS Lab VMs"
|
|
pveum pool modify IncusLab --storage local-lvm,local
|
|
pveum aclmod /pool/IncusLab -user automation@pve -role IncusOSDeploy
|
|
```
|
|
|
|
Add to your config:
|
|
|
|
```yaml
|
|
proxmox:
|
|
pool: IncusLab
|
|
```
|
|
|
|
Deploy and verify:
|
|
|
|
```bash
|
|
./incusos/incusos-proxmox --dry-run my-test.yaml # Should show pool in plan
|
|
./incusos/incusos-proxmox my-test.yaml # VM created in pool
|
|
./incusos/incusos-proxmox --status my-test.yaml # Status scoped to pool
|
|
```
|
|
|
|
### Step 11: (Optional) Full Lab Deployment
|
|
|
|
Once single-VM tests pass, deploy the full 4-VM lab:
|
|
|
|
```bash
|
|
cp incusos/examples/proxmox-lab.yaml my-lab.yaml
|
|
# Edit: change host IP, adjust start_vmid if needed
|
|
./incusos/incusos-proxmox --dry-run my-lab.yaml
|
|
./incusos/incusos-proxmox my-lab.yaml
|
|
```
|
|
|
|
This deploys 1 Operations Center + 3 Incus nodes. After deployment:
|
|
|
|
1. Access Operations Center via browser at its reported IP
|
|
2. On the init node, run `incus cluster add <node-name>` to get join tokens
|
|
3. Join the other two nodes using the tokens
|
|
|
|
---
|
|
|
|
## Lab Validation with lab-test
|
|
|
|
The `lab-test` script automates the end-to-end validation of deployed IncusOS
|
|
VMs: single-node workloads, cluster formation, migration, and evacuation.
|
|
|
|
### Quick Start: Cluster Testing
|
|
|
|
```bash
|
|
# 1. Check environment
|
|
./incusos/incusos-proxmox --doctor
|
|
|
|
# 2. Deploy 3 nodes
|
|
./incusos/incusos-proxmox --yes examples/lab-cluster.yaml
|
|
|
|
# 3. Add incus remotes for each node (use IPs from --status)
|
|
./incusos/incusos-proxmox --status examples/lab-cluster.yaml
|
|
incus remote add incus-lab-01 <IP1> --accept-certificate
|
|
incus remote add incus-lab-02 <IP2> --accept-certificate
|
|
incus remote add incus-lab-03 <IP3> --accept-certificate
|
|
|
|
# 4. Run full lab validation
|
|
./incusos/lab-test --yes examples/lab-cluster.yaml
|
|
```
|
|
|
|
### Phase-by-Phase Testing
|
|
|
|
```bash
|
|
# Single-node workloads only (container + VM launch/exec)
|
|
./incusos/lab-test --phase single --skip-deploy examples/lab-cluster.yaml
|
|
|
|
# Cluster formation only
|
|
./incusos/lab-test --phase cluster --skip-deploy examples/lab-cluster.yaml
|
|
|
|
# Cluster workloads (scheduler placement + targeted placement)
|
|
./incusos/lab-test --phase workload --skip-deploy examples/lab-cluster.yaml
|
|
|
|
# Migration and evacuation tests
|
|
./incusos/lab-test --phase migrate --skip-deploy examples/lab-cluster.yaml
|
|
```
|
|
|
|
### Manual Cluster Formation (Alternative)
|
|
|
|
If you prefer to form the cluster manually instead of using `lab-test`:
|
|
|
|
```bash
|
|
# Step 1: Enable clustering on first node
|
|
incus cluster enable incus-lab-01: incus-lab-01
|
|
|
|
# Step 2: Generate join token for second node
|
|
incus cluster add incus-lab-01: incus-lab-02
|
|
|
|
# Step 3: Join second node
|
|
incus cluster join incus-lab-01: incus-lab-02:
|
|
|
|
# Step 4: Repeat for third node
|
|
incus cluster add incus-lab-01: incus-lab-03
|
|
incus cluster join incus-lab-01: incus-lab-03:
|
|
|
|
# Step 5: Verify
|
|
incus cluster list incus-lab-01:
|
|
```
|
|
|
|
**Key points:**
|
|
- `apply_defaults: true` on all nodes ensures matching storage pool and
|
|
network bridge configurations (required for migration)
|
|
- After joining, manage the cluster through the init node's remote
|
|
- VM migration requires matching storage pool names across nodes
|
|
|
|
### Cleanup
|
|
|
|
```bash
|
|
# Remove test instances only (keeps VMs and cluster)
|
|
./incusos/lab-test --cleanup examples/lab-cluster.yaml
|
|
|
|
# Destroy VMs only
|
|
./incusos/incusos-proxmox --cleanup examples/lab-cluster.yaml
|
|
|
|
# Full cleanup: VMs + ISOs + seeds + remotes + cache
|
|
./incusos/incusos-proxmox --cleanup --deep --yes examples/lab-cluster.yaml
|
|
```
|
|
|
|
---
|
|
|
|
## Operations Center Lab
|
|
|
|
### Deploy
|
|
|
|
```bash
|
|
./incusos/incusos-proxmox --yes examples/lab-oc.yaml
|
|
./incusos/incusos-proxmox --status examples/lab-oc.yaml
|
|
```
|
|
|
|
### Access the Web UI
|
|
|
|
```bash
|
|
# Export client certificate for browser
|
|
openssl pkcs12 -export \
|
|
-inkey ~/.config/incus/client.key \
|
|
-in ~/.config/incus/client.crt \
|
|
-out client.pfx
|
|
|
|
# Import client.pfx into Firefox, navigate to https://<OC_IP>:8443
|
|
```
|
|
|
|
### OC CLI Setup
|
|
|
|
```bash
|
|
# Copy incus client certs for OC CLI
|
|
mkdir -p ~/.config/operations-center
|
|
cp ~/.config/incus/client.crt ~/.config/operations-center/
|
|
cp ~/.config/incus/client.key ~/.config/operations-center/
|
|
|
|
# Add OC remote
|
|
operations-center remote add lab-oc https://<OC_IP>:8443
|
|
operations-center remote switch lab-oc
|
|
|
|
# Check for provisioning updates
|
|
operations-center provisioning update list
|
|
```
|
|
|
|
### Provisioning Workflow
|
|
|
|
```bash
|
|
# Create a provisioning token
|
|
operations-center provisioning token add --description "lab nodes" --uses 10
|
|
|
|
# Get provisioned ISO
|
|
operations-center provisioning token get-image <token> ~/Downloads/IncusOS-provisioned.iso
|
|
|
|
# Redeploy nodes with provisioned ISO, or check if manual adoption works
|
|
```
|
|
|
|
**Note:** Operations Center is under active development (v0.2.2+). Commands and
|
|
APIs may change between versions. The `--doctor` command reports whether the OC
|
|
CLI is installed.
|
|
|
|
---
|
|
|
|
## Troubleshooting
|
|
|
|
| Symptom | Likely Cause | Fix |
|
|
|---------|-------------|-----|
|
|
| SSH connection refused | Key auth not set up | `ssh-copy-id root@<host>` |
|
|
| Storage pool not found | Wrong pool name in config | Check `pvesm status` on Proxmox |
|
|
| ISO download fails | CDN unreachable | Download manually, use `--iso path/to/file.iso` |
|
|
| VM creation fails | VMID collision | Use higher `start_vmid` or check `qm list` |
|
|
| Install times out (10 min) | Slow I/O or boot issue | Check VM console in Proxmox web UI |
|
|
| No IP reported | No guest agent (expected) | Check Proxmox console; script uses ARP-based detection |
|
|
| Wrong IP reported | Stale ARP entries | Script flushes ARP, but verify via Proxmox console |
|
|
| "install media detected" on boot | ide2/ide3 still attached | Script auto-detaches; if manual, delete ide2+ide3 then start |
|
|
| "no target device matched" | Explicit `disk-target` in seed | Remove `disk-target`; let IncusOS auto-detect |
|
|
| "field server not found" YAML error | Wrong preseed structure | Use `preseed.certificates[]` not `preseed.server.certificates[]` |
|
|
| Cert not auto-detected | No Incus keypair | Run `incus remote list` to generate one |
|
|
| `genisoimage` not found | genisoimage not installed | Install `genisoimage` package |
|
|
| Pool not found | Resource pool not created | Run `pveum pool add <name>` on Proxmox |
|
|
| Re-run destroys VMs | Used option 3 in reconcile | Use option 1 (checks) or option 2 (continue) instead |
|
|
| `--status` shows "not found" | VM was cleaned up | Deploy first, then check status |
|