Add lab validation tooling: doctor check, deep cleanup, lab-test script

Implements the first wave of lab validation infrastructure:

incusos-proxmox enhancements:
- --doctor flag: standalone environment health check (tool versions,
  IncusOS CDN availability, Proxmox connectivity). Works with or
  without a config file.
- --cleanup --deep: extended cleanup that also removes seed ISOs,
  IncusOS ISOs from Proxmox storage, matching incus remotes, and
  the local ISO cache (~/.cache/incusos-proxmox/iso-cache/).
- --cleanup-all: pool-wide cleanup that destroys ALL VMs with the
  [incusos-lab:managed] marker in the configured resource pool.
  Requires a pool to be set in config for safety scoping.

New script -- lab-test:
- Guided lab validation with 5 phases: deploy & verify, single-node
  workloads (container + VM lifecycle), cluster formation (enable +
  join via incus remotes), cluster workloads (scheduler-placed +
  targeted), and migration & evacuation (stop/move + evacuate/restore).
- Interactive by default with --yes for unattended runs.
- Reports PASS/FAIL/SKIP per test with a summary.
- Supports --dry-run, --skip-deploy, --cleanup, per-phase execution.
- Reads the same YAML config format as incusos-proxmox.

New example configs:
- lab-cluster.yaml: 3-node Incus cluster (VMID 900+) for testing
  clustering, workloads, and migration.
- lab-oc.yaml: Operations Center + 3 Incus nodes (VMID 910+) for
  testing OC-managed provisioning and cluster formation.

Documentation:
- CLAUDE.md: added clustering workflow (incus cluster enable/add/join
  via remotes), lab-test overview, Operations Center CLI notes,
  doctor and cleanup feature descriptions.
- README.md: added lab-test section with usage/phases/options, doctor
  check docs, deep cleanup docs, new example files in listing.
- TESTING.md: added full lab validation guide (quick start, phase-by-
  phase testing, manual cluster formation alternative, OC lab workflow
  with CLI setup and provisioning steps), cleanup instructions.

Not yet tested against live infrastructure -- next step is to deploy
lab-cluster.yaml, add incus remotes, and run lab-test end-to-end.
Known areas to validate: cluster join syntax with remotes, VM nested
virt support, evacuation behavior, apply_defaults consistency across
cluster members.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Maarten 2026-02-19 22:00:05 +01:00
parent a0a4b48c73
commit 2aca6fb33b
10 changed files with 2649 additions and 24 deletions

View File

@ -18,7 +18,8 @@ incu-contrib/
│ ├── incusos-iso # ISO/IMG builder (wraps flasher-tool) │ ├── incusos-iso # ISO/IMG builder (wraps flasher-tool)
│ ├── incusos-seed # Seed archive generator │ ├── incusos-seed # Seed archive generator
│ ├── incusos-proxmox # Declarative Proxmox VM deployment │ ├── incusos-proxmox # Declarative Proxmox VM deployment
│ ├── TESTING.md # Testing guide for incusos-proxmox │ ├── lab-test # Guided lab validation (workloads, clustering, migration)
│ ├── TESTING.md # Testing guide for incusos-proxmox and lab-test
│ └── examples/ # Example seed + Proxmox YAML files │ └── examples/ # Example seed + Proxmox YAML files
└── notes/ # Research notes and reference material └── notes/ # Research notes and reference material
``` ```
@ -116,6 +117,81 @@ incu-contrib/
IncusOS sits at "please remove installation media" and waits indefinitely. IncusOS sits at "please remove installation media" and waits indefinitely.
It does NOT halt automatically. `force_reboot` triggers a guest-level reboot It does NOT halt automatically. `force_reboot` triggers a guest-level reboot
(note: this does NOT reset QEMU VM uptime -- only a Proxmox stop/start does). (note: this does NOT reset QEMU VM uptime -- only a Proxmox stop/start does).
- **Resource pool isolation**: the optional `proxmox.pool` config field scopes
all VM operations to a Proxmox resource pool. When set, the script only
"sees" VMs in that pool (for collision detection and cleanup), and the API
token ACL can be scoped to `/pool/<name>` instead of `/`. Setup:
```
pveum pool add IncusLab --comment "IncusOS Lab VMs"
pveum pool modify IncusLab --storage local-lvm,local
pveum aclmod /pool/IncusLab -user automation@pve -role IncusOSDeploy
```
- **`--status` command**: `incusos-proxmox --status config.yaml` shows per-VM
deployment status (Proxmox state, install status, IP, port 8443, incus
remote). Also runs post-deployment checks (Incus connectivity, Operations
Center URL).
- **Reconcile on re-runs**: when `--phase all` detects existing VMs from
config, an interactive menu offers: (1) run status checks, (2) continue
install for incomplete VMs, (3) destroy and redeploy, (4) abort. With
`--yes`, defaults to option 1 (safe -- never auto-destroys).
- **Install idempotency**: `phase_install` checks each VM's state before
acting -- already-running VMs are skipped, stopped-but-installed VMs are
started from disk, only VMs with install media proceed through installation.
### Incus clustering via remotes
- **Cluster formation** is done entirely through the `incus` CLI using remotes.
No SSH to the IncusOS nodes is needed (IncusOS is immutable, no shell access).
- **Workflow:**
```bash
# Enable clustering on first node
incus cluster enable <remote1>: <member-name>
# Generate join token (on cluster, for new member name)
incus cluster add <remote1>: <new-member-name>
# Join second node to cluster
incus cluster join <remote1>: <remote2>:
# Verify
incus cluster list <remote1>:
```
- **`apply_defaults: true`** must be set on all nodes that will be clustered --
this ensures matching storage pool (`local`) and network bridge names, which
are required for migration.
- After joining, the cluster is managed through the init node's remote. The
individual node remotes still work for node-specific operations.
- **`lab-test`** automates cluster formation, workload testing, and migration.
### Lab validation (lab-test)
- **`lab-test`** reads the same YAML config as `incusos-proxmox` and operates
on the VM names defined there (expects `incus` remotes to exist).
- **Phases:** deploy, single (workloads), cluster, workload (on cluster),
migrate (stop/move + evacuate/restore).
- **Test instances** use names starting with `test-` for easy cleanup.
- The script reports PASS/FAIL/SKIP for each test and prints a summary.
### Operations Center
- **CLI binary:** `operations-center` (installed from GitHub releases or
built from source at `github.com/FuturFusion/operations-center`).
- **Config directory:** `~/.config/operations-center/` (uses same cert format
as Incus: copy `client.crt` and `client.key` from `~/.config/incus/`).
- **Provisioning workflow:** create token → get provisioned ISO → deploy nodes
with that ISO → nodes auto-register with OC → form cluster via OC.
- **OC is under active development** (v0.2.2+). Commands and APIs may change.
The `--doctor` command on `incusos-proxmox` reports whether the CLI is
installed.
### incusos-proxmox doctor and cleanup
- **`--doctor`**: standalone environment check. No config file required.
Checks tool versions, IncusOS CDN, and optionally Proxmox connectivity.
- **`--cleanup --deep`**: destroys VMs + deletes ISOs/seeds from Proxmox +
removes incus remotes + clears local ISO cache.
- **`--cleanup-all`**: pool-wide cleanup. Destroys all VMs with the
`[incusos-lab:managed]` marker in the configured resource pool.
## Coding conventions for scripts ## Coding conventions for scripts

View File

@ -9,6 +9,7 @@ Scripts for building IncusOS installation media and seed configurations.
| [`incusos-iso`](#incusos-iso) | Build customized IncusOS ISO/IMG images using the flasher-tool | | [`incusos-iso`](#incusos-iso) | Build customized IncusOS ISO/IMG images using the flasher-tool |
| [`incusos-seed`](#incusos-seed) | Generate seed archives (tar, FAT, or ISO 9660) for installation customization | | [`incusos-seed`](#incusos-seed) | Generate seed archives (tar, FAT, or ISO 9660) for installation customization |
| [`incusos-proxmox`](#incusos-proxmox) | Declarative IncusOS deployment on Proxmox VE | | [`incusos-proxmox`](#incusos-proxmox) | Declarative IncusOS deployment on Proxmox VE |
| [`lab-test`](#lab-test) | Guided lab validation (workloads, clustering, migration) |
All scripts support automatic client certificate injection from the local Incus All scripts support automatic client certificate injection from the local Incus
installation, so the freshly installed system is immediately manageable. installation, so the freshly installed system is immediately manageable.
@ -274,6 +275,9 @@ This script ensures all settings are correct, preventing install failures.
# Deploy everything # Deploy everything
./incusos-proxmox examples/proxmox-lab.yaml ./incusos-proxmox examples/proxmox-lab.yaml
# Check deployment status
./incusos-proxmox --status examples/proxmox-lab.yaml
# Deploy a single VM from the config # Deploy a single VM from the config
./incusos-proxmox --vm ops-center examples/proxmox-lab.yaml ./incusos-proxmox --vm ops-center examples/proxmox-lab.yaml
@ -284,7 +288,7 @@ This script ensures all settings are correct, preventing install failures.
### Options Reference ### Options Reference
``` ```
-p, --phase PHASE validate, prepare, upload, create, install, all (default) -p, --phase PHASE validate, prepare, upload, create, install, status, all (default)
-m, --method METHOD Override connection method: ssh (default), api -m, --method METHOD Override connection method: ssh (default), api
-H, --host HOST Override Proxmox host from config -H, --host HOST Override Proxmox host from config
-i, --iso FILE Use local ISO (skip CDN download) -i, --iso FILE Use local ISO (skip CDN download)
@ -292,13 +296,55 @@ This script ensures all settings are correct, preventing install failures.
--no-cert Skip certificate injection --no-cert Skip certificate injection
--vm NAME Operate on a single VM only --vm NAME Operate on a single VM only
-y, --yes Skip confirmation prompts -y, --yes Skip confirmation prompts
--doctor Check tool versions and environment health
--status Check deployment status (shortcut for --phase status)
--cleanup Destroy VMs defined in config --cleanup Destroy VMs defined in config
--cleanup --deep Also remove ISOs, seeds, remotes, and local cache
--cleanup-all Destroy ALL managed VMs in the resource pool
--force-cleanup Destroy VMs even without management marker
--dry-run Preview without executing --dry-run Preview without executing
-q, --quiet Suppress info output -q, --quiet Suppress info output
-h, --help Full help text -h, --help Full help text
-V, --version Show version -V, --version Show version
``` ```
### Doctor Check
Run `--doctor` to verify your environment without a config file:
```bash
./incusos-proxmox --doctor
```
This checks: incus client, flasher-tool, operations-center CLI (optional),
python3, curl, genisoimage, jq, incusos-seed sibling, and IncusOS CDN
availability.
If a config file is also provided, additionally checks Proxmox connectivity,
pool existence, and API token validity:
```bash
./incusos-proxmox --doctor examples/lab-cluster.yaml
```
### Deep Cleanup
Use `--deep` with `--cleanup` or `--cleanup-all` to remove all artifacts:
```bash
# VMs + ISOs + seeds + incus remotes + local cache
./incusos-proxmox --cleanup --deep examples/lab-cluster.yaml
# Pool-wide: destroy all managed VMs + artifacts
./incusos-proxmox --cleanup-all --deep examples/lab-cluster.yaml
```
`--deep` additionally:
- Deletes seed ISOs from Proxmox storage
- Deletes IncusOS ISOs from Proxmox storage
- Removes matching `incus remote` entries
- Clears the local ISO cache (`~/.cache/incusos-proxmox/iso-cache/`)
### Deployment Phases ### Deployment Phases
| Phase | What it does | | Phase | What it does |
@ -308,6 +354,7 @@ This script ensures all settings are correct, preventing install failures.
| `upload` | Upload ISO and seed images to Proxmox ISO storage | | `upload` | Upload ISO and seed images to Proxmox ISO storage |
| `create` | Create VMs with IncusOS-correct settings (UEFI, TPM, VirtIO-SCSI, etc.) | | `create` | Create VMs with IncusOS-correct settings (UEFI, TPM, VirtIO-SCSI, etc.) |
| `install` | Start VMs, detect install completion via disk I/O monitoring, detach CD-ROMs, boot from disk | | `install` | Start VMs, detect install completion via disk I/O monitoring, detach CD-ROMs, boot from disk |
| `status` | Check deployment status: Proxmox state, IP, port 8443, incus remotes, post-deploy checks |
### Config File Format ### Config File Format
@ -318,6 +365,7 @@ proxmox:
storage: local-lvm # VM disk storage (default: local-lvm) storage: local-lvm # VM disk storage (default: local-lvm)
iso_storage: local # ISO storage (default: local) iso_storage: local # ISO storage (default: local)
bridge: vmbr0 # Network bridge (default: vmbr0) bridge: vmbr0 # Network bridge (default: vmbr0)
pool: IncusLab # Resource pool (optional, for isolation)
method: ssh # ssh (default) or api method: ssh # ssh (default) or api
ssh_user: root # For ssh method ssh_user: root # For ssh method
@ -364,6 +412,54 @@ pveum user token add automation@pve deploy --privsep 0
export PROXMOX_TOKEN_SECRET="<secret-from-above>" export PROXMOX_TOKEN_SECRET="<secret-from-above>"
``` ```
### Resource Pool Isolation
For tighter access control, create a Proxmox resource pool and scope the API
token to it instead of `/`:
```bash
# On Proxmox (one-time setup)
pveum pool add IncusLab --comment "IncusOS Lab VMs"
pveum pool modify IncusLab --storage local-lvm,local
pveum aclmod /pool/IncusLab -user automation@pve -role IncusOSDeploy
```
Then in your config file:
```yaml
proxmox:
pool: IncusLab
```
When a pool is configured, the script only "sees" VMs in that pool — both for
collision detection and cleanup. The VM management marker includes the pool name
for traceability.
### Re-runs, Status, and Reconcile
**Status check:** Use `--status` to verify deployment health without modifying anything:
```bash
./incusos-proxmox --status examples/proxmox-lab.yaml
```
Shows per-VM: Proxmox state, install status, IP address, port 8443 reachability,
and incus remote connectivity.
**Re-run safety:** When re-running a full deployment (`--phase all`), the script
detects existing VMs and presents an interactive menu:
1. Run status checks and post-deployment verification
2. Continue deployment (skip existing, install incomplete VMs)
3. Destroy and redeploy cleanly
4. Abort
With `--yes`, defaults to option 1 (safe — never auto-destroys).
**Install idempotency:** The install phase checks each VM's state before acting:
already-running VMs are skipped, stopped-but-installed VMs are started from disk,
and only VMs with install media still attached proceed through installation.
### Proxmox VM Settings ### Proxmox VM Settings
Every VM is created with these IncusOS-specific settings: Every VM is created with these IncusOS-specific settings:
@ -393,6 +489,60 @@ Every VM is created with these IncusOS-specific settings:
- [`examples/proxmox-lab.yaml`](examples/proxmox-lab.yaml) -- Full 4-VM lab (ops-center + 3-node cluster) - [`examples/proxmox-lab.yaml`](examples/proxmox-lab.yaml) -- Full 4-VM lab (ops-center + 3-node cluster)
- [`examples/proxmox-minimal.yaml`](examples/proxmox-minimal.yaml) -- Single standalone Incus VM - [`examples/proxmox-minimal.yaml`](examples/proxmox-minimal.yaml) -- Single standalone Incus VM
## lab-test
Guided lab validation script for IncusOS deployments. Walks through workload
testing, cluster formation, and migration tests step by step. Interactive by
default — prints what it will do, confirms, executes, and reports pass/fail.
### Quick Start
```bash
# Run all phases (deploy, single-node tests, cluster, workloads, migration)
./incusos/lab-test examples/lab-cluster.yaml
# Deploy and verify only
./incusos/lab-test --phase deploy examples/lab-cluster.yaml
# Single-node workload tests
./incusos/lab-test --phase single examples/lab-cluster.yaml
# Form cluster from deployed nodes
./incusos/lab-test --phase cluster examples/lab-cluster.yaml
# Test migration and evacuation
./incusos/lab-test --phase migrate examples/lab-cluster.yaml
# Full run without prompts
./incusos/lab-test --yes examples/lab-cluster.yaml
# Clean up test instances (keeps VMs)
./incusos/lab-test --cleanup examples/lab-cluster.yaml
```
### Phases
| Phase | What it does |
|-------|-------------|
| `deploy` | Deploy VMs (via incusos-proxmox), verify incus remotes, check connectivity |
| `single` | Launch a test container and VM on the first node, exec, cleanup |
| `cluster` | Enable clustering on node 1, join remaining nodes, verify membership |
| `workload` | Launch instances on the cluster (scheduler-placed + targeted), cleanup |
| `migrate` | Stop/move container between nodes, evacuate/restore a node |
### Options
```
-p, --phase PHASE deploy, single, cluster, workload, migrate, all (default)
--skip-deploy Skip deploy/status check (assume VMs already running)
--cleanup Remove test instances without destroying VMs
-y, --yes Skip confirmation prompts
--dry-run Preview actions
-q, --quiet Suppress info output
-h, --help Full help text
-V, --version Show version
```
## Client Certificate Auto-Detection ## Client Certificate Auto-Detection
Both scripts automatically look for a client certificate in this order: Both scripts automatically look for a client certificate in this order:
@ -454,6 +604,8 @@ See the [`examples/`](examples/) directory for sample seed configurations:
- [`network-static.yaml`](examples/network-static.yaml) -- Static network configuration - [`network-static.yaml`](examples/network-static.yaml) -- Static network configuration
- [`proxmox-lab.yaml`](examples/proxmox-lab.yaml) -- Proxmox: 4-VM lab (ops-center + 3-node cluster) - [`proxmox-lab.yaml`](examples/proxmox-lab.yaml) -- Proxmox: 4-VM lab (ops-center + 3-node cluster)
- [`proxmox-minimal.yaml`](examples/proxmox-minimal.yaml) -- Proxmox: single standalone VM - [`proxmox-minimal.yaml`](examples/proxmox-minimal.yaml) -- Proxmox: single standalone VM
- [`lab-cluster.yaml`](examples/lab-cluster.yaml) -- Proxmox: 3-node cluster for lab validation
- [`lab-oc.yaml`](examples/lab-oc.yaml) -- Proxmox: Operations Center + 3 nodes
## Security Notes ## Security Notes

View File

@ -196,7 +196,23 @@ Using VMID 900+ avoids clashing with existing VMs.
disk, and reboot disk, and reboot
- After reboot you should see the IncusOS login prompt - After reboot you should see the IncusOS login prompt
### Step 5: Verify the Deployed VM ### Step 5: Check Deployment Status
After the deploy completes, verify with the built-in status command:
```bash
./incusos/incusos-proxmox --status my-test.yaml
```
**Verify in the output:**
- VM `incus-test` exists and is managed
- Status is `running`, install state is `installed`
- IP address detected
- Port 8443 is open
- Incus remote status shown (if configured)
### Step 6: Verify the Deployed VM
Once the script reports success with an IP address: Once the script reports success with an IP address:
@ -217,7 +233,26 @@ incus network list incus-test:
ARP-based lookup (MAC from Proxmox VM config → ARP table). If the IP is wrong, ARP-based lookup (MAC from Proxmox VM config → ARP table). If the IP is wrong,
check the Proxmox console — stale ARP entries can be misleading. check the Proxmox console — stale ARP entries can be misleading.
### Step 6: Clean Up the Test VM ### Step 7: Test Re-run Reconcile
Re-run the deploy to verify the reconcile menu:
```bash
./incusos/incusos-proxmox my-test.yaml
```
**Verify:** The script detects the existing VM and shows the interactive menu.
Choose option 1 to run status checks without modifying anything.
Also test with `--yes` to verify it defaults to safe behavior:
```bash
./incusos/incusos-proxmox --yes my-test.yaml
```
**Verify:** Runs post-deployment checks, does NOT destroy.
### Step 8: Clean Up the Test VM
```bash ```bash
./incusos/incusos-proxmox --cleanup my-test.yaml ./incusos/incusos-proxmox --cleanup my-test.yaml
@ -234,7 +269,7 @@ Remove the remote too:
incus remote remove incus-test incus remote remove incus-test
``` ```
### Step 7: (Optional) Test API Method ### Step 9: (Optional) Test API Method
After single-VM success with SSH, switch to API tokens. After single-VM success with SSH, switch to API tokens.
@ -269,7 +304,34 @@ proxmox:
Repeat steps 3-6 using the API method. Repeat steps 3-6 using the API method.
### Step 8: (Optional) Full Lab Deployment ### Step 10: (Optional) Test with Resource Pool
For API method with pool isolation:
**On Proxmox:**
```bash
pveum pool add IncusLab --comment "IncusOS Lab VMs"
pveum pool modify IncusLab --storage local-lvm,local
pveum aclmod /pool/IncusLab -user automation@pve -role IncusOSDeploy
```
Add to your config:
```yaml
proxmox:
pool: IncusLab
```
Deploy and verify:
```bash
./incusos/incusos-proxmox --dry-run my-test.yaml # Should show pool in plan
./incusos/incusos-proxmox my-test.yaml # VM created in pool
./incusos/incusos-proxmox --status my-test.yaml # Status scoped to pool
```
### Step 11: (Optional) Full Lab Deployment
Once single-VM tests pass, deploy the full 4-VM lab: Once single-VM tests pass, deploy the full 4-VM lab:
@ -288,6 +350,144 @@ This deploys 1 Operations Center + 3 Incus nodes. After deployment:
--- ---
## Lab Validation with lab-test
The `lab-test` script automates the end-to-end validation of deployed IncusOS
VMs: single-node workloads, cluster formation, migration, and evacuation.
### Quick Start: Cluster Testing
```bash
# 1. Check environment
./incusos/incusos-proxmox --doctor
# 2. Deploy 3 nodes
./incusos/incusos-proxmox --yes examples/lab-cluster.yaml
# 3. Add incus remotes for each node (use IPs from --status)
./incusos/incusos-proxmox --status examples/lab-cluster.yaml
incus remote add incus-lab-01 <IP1> --accept-certificate
incus remote add incus-lab-02 <IP2> --accept-certificate
incus remote add incus-lab-03 <IP3> --accept-certificate
# 4. Run full lab validation
./incusos/lab-test --yes examples/lab-cluster.yaml
```
### Phase-by-Phase Testing
```bash
# Single-node workloads only (container + VM launch/exec)
./incusos/lab-test --phase single --skip-deploy examples/lab-cluster.yaml
# Cluster formation only
./incusos/lab-test --phase cluster --skip-deploy examples/lab-cluster.yaml
# Cluster workloads (scheduler placement + targeted placement)
./incusos/lab-test --phase workload --skip-deploy examples/lab-cluster.yaml
# Migration and evacuation tests
./incusos/lab-test --phase migrate --skip-deploy examples/lab-cluster.yaml
```
### Manual Cluster Formation (Alternative)
If you prefer to form the cluster manually instead of using `lab-test`:
```bash
# Step 1: Enable clustering on first node
incus cluster enable incus-lab-01: incus-lab-01
# Step 2: Generate join token for second node
incus cluster add incus-lab-01: incus-lab-02
# Step 3: Join second node
incus cluster join incus-lab-01: incus-lab-02:
# Step 4: Repeat for third node
incus cluster add incus-lab-01: incus-lab-03
incus cluster join incus-lab-01: incus-lab-03:
# Step 5: Verify
incus cluster list incus-lab-01:
```
**Key points:**
- `apply_defaults: true` on all nodes ensures matching storage pool and
network bridge configurations (required for migration)
- After joining, manage the cluster through the init node's remote
- VM migration requires matching storage pool names across nodes
### Cleanup
```bash
# Remove test instances only (keeps VMs and cluster)
./incusos/lab-test --cleanup examples/lab-cluster.yaml
# Destroy VMs only
./incusos/incusos-proxmox --cleanup examples/lab-cluster.yaml
# Full cleanup: VMs + ISOs + seeds + remotes + cache
./incusos/incusos-proxmox --cleanup --deep --yes examples/lab-cluster.yaml
```
---
## Operations Center Lab
### Deploy
```bash
./incusos/incusos-proxmox --yes examples/lab-oc.yaml
./incusos/incusos-proxmox --status examples/lab-oc.yaml
```
### Access the Web UI
```bash
# Export client certificate for browser
openssl pkcs12 -export \
-inkey ~/.config/incus/client.key \
-in ~/.config/incus/client.crt \
-out client.pfx
# Import client.pfx into Firefox, navigate to https://<OC_IP>:8443
```
### OC CLI Setup
```bash
# Copy incus client certs for OC CLI
mkdir -p ~/.config/operations-center
cp ~/.config/incus/client.crt ~/.config/operations-center/
cp ~/.config/incus/client.key ~/.config/operations-center/
# Add OC remote
operations-center remote add lab-oc https://<OC_IP>:8443
operations-center remote switch lab-oc
# Check for provisioning updates
operations-center provisioning update list
```
### Provisioning Workflow
```bash
# Create a provisioning token
operations-center provisioning token add --description "lab nodes" --uses 10
# Get provisioned ISO
operations-center provisioning token get-image <token> ~/Downloads/IncusOS-provisioned.iso
# Redeploy nodes with provisioned ISO, or check if manual adoption works
```
**Note:** Operations Center is under active development (v0.2.2+). Commands and
APIs may change between versions. The `--doctor` command reports whether the OC
CLI is installed.
---
## Troubleshooting ## Troubleshooting
| Symptom | Likely Cause | Fix | | Symptom | Likely Cause | Fix |
@ -304,3 +504,6 @@ This deploys 1 Operations Center + 3 Incus nodes. After deployment:
| "field server not found" YAML error | Wrong preseed structure | Use `preseed.certificates[]` not `preseed.server.certificates[]` | | "field server not found" YAML error | Wrong preseed structure | Use `preseed.certificates[]` not `preseed.server.certificates[]` |
| Cert not auto-detected | No Incus keypair | Run `incus remote list` to generate one | | Cert not auto-detected | No Incus keypair | Run `incus remote list` to generate one |
| `genisoimage` not found | genisoimage not installed | Install `genisoimage` package | | `genisoimage` not found | genisoimage not installed | Install `genisoimage` package |
| Pool not found | Resource pool not created | Run `pveum pool add <name>` on Proxmox |
| Re-run destroys VMs | Used option 3 in reconcile | Use option 1 (checks) or option 2 (continue) instead |
| `--status` shows "not found" | VM was cleaned up | Deploy first, then check status |

View File

@ -0,0 +1,44 @@
# lab-cluster.yaml - 3-node IncusOS cluster for lab testing
#
# Deploys 3 standalone Incus nodes that can be clustered after deployment.
# All nodes get apply_defaults: true so they have matching storage pool
# and network bridge configurations.
#
# Usage:
# incusos-proxmox --dry-run examples/lab-cluster.yaml # Preview
# incusos-proxmox --yes examples/lab-cluster.yaml # Deploy
# incusos-proxmox --status examples/lab-cluster.yaml # Check status
# incusos-proxmox --cleanup examples/lab-cluster.yaml # Tear down VMs
# incusos-proxmox --cleanup --deep examples/lab-cluster.yaml # Full cleanup
#
# After deployment, form a cluster with lab-test:
# lab-test --phase cluster examples/lab-cluster.yaml
proxmox:
host: 192.168.1.29
method: api
api_token_id: automation@pve!deploy
storage: local-zfs
pool: IncusLab
defaults:
cores: 2
memory: 4096
disk: 50
start_vmid: 900
vms:
- name: incus-lab-01
app: incus
apply_defaults: true
cores: 4
memory: 8192
disk: 64
- name: incus-lab-02
app: incus
apply_defaults: true
- name: incus-lab-03
app: incus
apply_defaults: true

View File

@ -0,0 +1,50 @@
# lab-oc.yaml - Operations Center + 3-node lab
#
# Deploys 1 Operations Center instance and 3 Incus nodes.
# Uses VMID range 910+ to avoid collisions with lab-cluster.yaml (900+).
#
# Workflow:
# 1. Deploy with incusos-proxmox
# 2. Access OC web UI (import client.pfx into browser)
# 3. Create provisioning token in OC
# 4. Either redeploy nodes with OC-provisioned ISO or manually adopt
# 5. Form cluster via OC CLI or web UI
#
# Usage:
# incusos-proxmox --dry-run examples/lab-oc.yaml
# incusos-proxmox --yes examples/lab-oc.yaml
# incusos-proxmox --status examples/lab-oc.yaml
# incusos-proxmox --cleanup --deep examples/lab-oc.yaml
proxmox:
host: 192.168.1.29
method: api
api_token_id: automation@pve!deploy
storage: local-zfs
pool: IncusLab
defaults:
cores: 2
memory: 4096
disk: 50
start_vmid: 910
vms:
- name: lab-oc
app: operations-center
apply_defaults: true
- name: lab-node-01
app: incus
apply_defaults: true
cores: 4
memory: 8192
disk: 64
- name: lab-node-02
app: incus
apply_defaults: true
- name: lab-node-03
app: incus
apply_defaults: true

View File

@ -3,15 +3,27 @@ proxmox:
method: api method: api
api_token_id: automation@pve!deploy api_token_id: automation@pve!deploy
storage: local-zfs # VM disk storage (default: local-lvm) storage: local-zfs # VM disk storage (default: local-lvm)
pool: IncusLab
defaults: defaults:
cores: 2
memory: 4096 # Minimum for IncusOS
disk: 50 # Minimum for IncusOS
start_vmid: 900 # High range to avoid collisions start_vmid: 900 # High range to avoid collisions
vms: vms:
- name: incus-test - name: incus-lab-01
app: incus app: incus
apply_defaults: true apply_defaults: true
cores: 4 cores: 4
memory: 8192 memory: 8192
disk: 64 disk: 64
- name: incus-lab-02
app: incus
apply_defaults: true
- name: ops-center
app: operations-center
apply_defaults: true

View File

@ -15,6 +15,7 @@ proxmox:
storage: local-lvm # VM disk storage storage: local-lvm # VM disk storage
iso_storage: local # ISO/image storage iso_storage: local # ISO/image storage
bridge: vmbr0 # Network bridge bridge: vmbr0 # Network bridge
# pool: IncusLab # Resource pool (optional, for isolation)
method: ssh # ssh (default) or api method: ssh # ssh (default) or api
ssh_user: root # SSH user for ssh method ssh_user: root # SSH user for ssh method
# api_token_id: automation@pve!deploy # For api method # api_token_id: automation@pve!deploy # For api method

View File

@ -8,6 +8,7 @@
proxmox: proxmox:
host: 192.168.1.10 # Change to your Proxmox host host: 192.168.1.10 # Change to your Proxmox host
# pool: IncusLab # Resource pool (optional, for isolation)
# All other settings use defaults: # All other settings use defaults:
# node: pve, storage: local-lvm, iso_storage: local, # node: pve, storage: local-lvm, iso_storage: local,
# bridge: vmbr0, method: ssh, ssh_user: root # bridge: vmbr0, method: ssh, ssh_user: root

File diff suppressed because it is too large Load Diff

1041
incusos/lab-test Executable file

File diff suppressed because it is too large Load Diff