Add lab validation tooling: doctor check, deep cleanup, lab-test script
Implements the first wave of lab validation infrastructure: incusos-proxmox enhancements: - --doctor flag: standalone environment health check (tool versions, IncusOS CDN availability, Proxmox connectivity). Works with or without a config file. - --cleanup --deep: extended cleanup that also removes seed ISOs, IncusOS ISOs from Proxmox storage, matching incus remotes, and the local ISO cache (~/.cache/incusos-proxmox/iso-cache/). - --cleanup-all: pool-wide cleanup that destroys ALL VMs with the [incusos-lab:managed] marker in the configured resource pool. Requires a pool to be set in config for safety scoping. New script -- lab-test: - Guided lab validation with 5 phases: deploy & verify, single-node workloads (container + VM lifecycle), cluster formation (enable + join via incus remotes), cluster workloads (scheduler-placed + targeted), and migration & evacuation (stop/move + evacuate/restore). - Interactive by default with --yes for unattended runs. - Reports PASS/FAIL/SKIP per test with a summary. - Supports --dry-run, --skip-deploy, --cleanup, per-phase execution. - Reads the same YAML config format as incusos-proxmox. New example configs: - lab-cluster.yaml: 3-node Incus cluster (VMID 900+) for testing clustering, workloads, and migration. - lab-oc.yaml: Operations Center + 3 Incus nodes (VMID 910+) for testing OC-managed provisioning and cluster formation. Documentation: - CLAUDE.md: added clustering workflow (incus cluster enable/add/join via remotes), lab-test overview, Operations Center CLI notes, doctor and cleanup feature descriptions. - README.md: added lab-test section with usage/phases/options, doctor check docs, deep cleanup docs, new example files in listing. - TESTING.md: added full lab validation guide (quick start, phase-by- phase testing, manual cluster formation alternative, OC lab workflow with CLI setup and provisioning steps), cleanup instructions. Not yet tested against live infrastructure -- next step is to deploy lab-cluster.yaml, add incus remotes, and run lab-test end-to-end. Known areas to validate: cluster join syntax with remotes, VM nested virt support, evacuation behavior, apply_defaults consistency across cluster members. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
a0a4b48c73
commit
2aca6fb33b
78
CLAUDE.md
78
CLAUDE.md
|
|
@ -18,7 +18,8 @@ incu-contrib/
|
||||||
│ ├── incusos-iso # ISO/IMG builder (wraps flasher-tool)
|
│ ├── incusos-iso # ISO/IMG builder (wraps flasher-tool)
|
||||||
│ ├── incusos-seed # Seed archive generator
|
│ ├── incusos-seed # Seed archive generator
|
||||||
│ ├── incusos-proxmox # Declarative Proxmox VM deployment
|
│ ├── incusos-proxmox # Declarative Proxmox VM deployment
|
||||||
│ ├── TESTING.md # Testing guide for incusos-proxmox
|
│ ├── lab-test # Guided lab validation (workloads, clustering, migration)
|
||||||
|
│ ├── TESTING.md # Testing guide for incusos-proxmox and lab-test
|
||||||
│ └── examples/ # Example seed + Proxmox YAML files
|
│ └── examples/ # Example seed + Proxmox YAML files
|
||||||
└── notes/ # Research notes and reference material
|
└── notes/ # Research notes and reference material
|
||||||
```
|
```
|
||||||
|
|
@ -116,6 +117,81 @@ incu-contrib/
|
||||||
IncusOS sits at "please remove installation media" and waits indefinitely.
|
IncusOS sits at "please remove installation media" and waits indefinitely.
|
||||||
It does NOT halt automatically. `force_reboot` triggers a guest-level reboot
|
It does NOT halt automatically. `force_reboot` triggers a guest-level reboot
|
||||||
(note: this does NOT reset QEMU VM uptime -- only a Proxmox stop/start does).
|
(note: this does NOT reset QEMU VM uptime -- only a Proxmox stop/start does).
|
||||||
|
- **Resource pool isolation**: the optional `proxmox.pool` config field scopes
|
||||||
|
all VM operations to a Proxmox resource pool. When set, the script only
|
||||||
|
"sees" VMs in that pool (for collision detection and cleanup), and the API
|
||||||
|
token ACL can be scoped to `/pool/<name>` instead of `/`. Setup:
|
||||||
|
```
|
||||||
|
pveum pool add IncusLab --comment "IncusOS Lab VMs"
|
||||||
|
pveum pool modify IncusLab --storage local-lvm,local
|
||||||
|
pveum aclmod /pool/IncusLab -user automation@pve -role IncusOSDeploy
|
||||||
|
```
|
||||||
|
- **`--status` command**: `incusos-proxmox --status config.yaml` shows per-VM
|
||||||
|
deployment status (Proxmox state, install status, IP, port 8443, incus
|
||||||
|
remote). Also runs post-deployment checks (Incus connectivity, Operations
|
||||||
|
Center URL).
|
||||||
|
- **Reconcile on re-runs**: when `--phase all` detects existing VMs from
|
||||||
|
config, an interactive menu offers: (1) run status checks, (2) continue
|
||||||
|
install for incomplete VMs, (3) destroy and redeploy, (4) abort. With
|
||||||
|
`--yes`, defaults to option 1 (safe -- never auto-destroys).
|
||||||
|
- **Install idempotency**: `phase_install` checks each VM's state before
|
||||||
|
acting -- already-running VMs are skipped, stopped-but-installed VMs are
|
||||||
|
started from disk, only VMs with install media proceed through installation.
|
||||||
|
|
||||||
|
### Incus clustering via remotes
|
||||||
|
|
||||||
|
- **Cluster formation** is done entirely through the `incus` CLI using remotes.
|
||||||
|
No SSH to the IncusOS nodes is needed (IncusOS is immutable, no shell access).
|
||||||
|
- **Workflow:**
|
||||||
|
```bash
|
||||||
|
# Enable clustering on first node
|
||||||
|
incus cluster enable <remote1>: <member-name>
|
||||||
|
|
||||||
|
# Generate join token (on cluster, for new member name)
|
||||||
|
incus cluster add <remote1>: <new-member-name>
|
||||||
|
|
||||||
|
# Join second node to cluster
|
||||||
|
incus cluster join <remote1>: <remote2>:
|
||||||
|
|
||||||
|
# Verify
|
||||||
|
incus cluster list <remote1>:
|
||||||
|
```
|
||||||
|
- **`apply_defaults: true`** must be set on all nodes that will be clustered --
|
||||||
|
this ensures matching storage pool (`local`) and network bridge names, which
|
||||||
|
are required for migration.
|
||||||
|
- After joining, the cluster is managed through the init node's remote. The
|
||||||
|
individual node remotes still work for node-specific operations.
|
||||||
|
- **`lab-test`** automates cluster formation, workload testing, and migration.
|
||||||
|
|
||||||
|
### Lab validation (lab-test)
|
||||||
|
|
||||||
|
- **`lab-test`** reads the same YAML config as `incusos-proxmox` and operates
|
||||||
|
on the VM names defined there (expects `incus` remotes to exist).
|
||||||
|
- **Phases:** deploy, single (workloads), cluster, workload (on cluster),
|
||||||
|
migrate (stop/move + evacuate/restore).
|
||||||
|
- **Test instances** use names starting with `test-` for easy cleanup.
|
||||||
|
- The script reports PASS/FAIL/SKIP for each test and prints a summary.
|
||||||
|
|
||||||
|
### Operations Center
|
||||||
|
|
||||||
|
- **CLI binary:** `operations-center` (installed from GitHub releases or
|
||||||
|
built from source at `github.com/FuturFusion/operations-center`).
|
||||||
|
- **Config directory:** `~/.config/operations-center/` (uses same cert format
|
||||||
|
as Incus: copy `client.crt` and `client.key` from `~/.config/incus/`).
|
||||||
|
- **Provisioning workflow:** create token → get provisioned ISO → deploy nodes
|
||||||
|
with that ISO → nodes auto-register with OC → form cluster via OC.
|
||||||
|
- **OC is under active development** (v0.2.2+). Commands and APIs may change.
|
||||||
|
The `--doctor` command on `incusos-proxmox` reports whether the CLI is
|
||||||
|
installed.
|
||||||
|
|
||||||
|
### incusos-proxmox doctor and cleanup
|
||||||
|
|
||||||
|
- **`--doctor`**: standalone environment check. No config file required.
|
||||||
|
Checks tool versions, IncusOS CDN, and optionally Proxmox connectivity.
|
||||||
|
- **`--cleanup --deep`**: destroys VMs + deletes ISOs/seeds from Proxmox +
|
||||||
|
removes incus remotes + clears local ISO cache.
|
||||||
|
- **`--cleanup-all`**: pool-wide cleanup. Destroys all VMs with the
|
||||||
|
`[incusos-lab:managed]` marker in the configured resource pool.
|
||||||
|
|
||||||
## Coding conventions for scripts
|
## Coding conventions for scripts
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -9,6 +9,7 @@ Scripts for building IncusOS installation media and seed configurations.
|
||||||
| [`incusos-iso`](#incusos-iso) | Build customized IncusOS ISO/IMG images using the flasher-tool |
|
| [`incusos-iso`](#incusos-iso) | Build customized IncusOS ISO/IMG images using the flasher-tool |
|
||||||
| [`incusos-seed`](#incusos-seed) | Generate seed archives (tar, FAT, or ISO 9660) for installation customization |
|
| [`incusos-seed`](#incusos-seed) | Generate seed archives (tar, FAT, or ISO 9660) for installation customization |
|
||||||
| [`incusos-proxmox`](#incusos-proxmox) | Declarative IncusOS deployment on Proxmox VE |
|
| [`incusos-proxmox`](#incusos-proxmox) | Declarative IncusOS deployment on Proxmox VE |
|
||||||
|
| [`lab-test`](#lab-test) | Guided lab validation (workloads, clustering, migration) |
|
||||||
|
|
||||||
All scripts support automatic client certificate injection from the local Incus
|
All scripts support automatic client certificate injection from the local Incus
|
||||||
installation, so the freshly installed system is immediately manageable.
|
installation, so the freshly installed system is immediately manageable.
|
||||||
|
|
@ -274,6 +275,9 @@ This script ensures all settings are correct, preventing install failures.
|
||||||
# Deploy everything
|
# Deploy everything
|
||||||
./incusos-proxmox examples/proxmox-lab.yaml
|
./incusos-proxmox examples/proxmox-lab.yaml
|
||||||
|
|
||||||
|
# Check deployment status
|
||||||
|
./incusos-proxmox --status examples/proxmox-lab.yaml
|
||||||
|
|
||||||
# Deploy a single VM from the config
|
# Deploy a single VM from the config
|
||||||
./incusos-proxmox --vm ops-center examples/proxmox-lab.yaml
|
./incusos-proxmox --vm ops-center examples/proxmox-lab.yaml
|
||||||
|
|
||||||
|
|
@ -284,7 +288,7 @@ This script ensures all settings are correct, preventing install failures.
|
||||||
### Options Reference
|
### Options Reference
|
||||||
|
|
||||||
```
|
```
|
||||||
-p, --phase PHASE validate, prepare, upload, create, install, all (default)
|
-p, --phase PHASE validate, prepare, upload, create, install, status, all (default)
|
||||||
-m, --method METHOD Override connection method: ssh (default), api
|
-m, --method METHOD Override connection method: ssh (default), api
|
||||||
-H, --host HOST Override Proxmox host from config
|
-H, --host HOST Override Proxmox host from config
|
||||||
-i, --iso FILE Use local ISO (skip CDN download)
|
-i, --iso FILE Use local ISO (skip CDN download)
|
||||||
|
|
@ -292,13 +296,55 @@ This script ensures all settings are correct, preventing install failures.
|
||||||
--no-cert Skip certificate injection
|
--no-cert Skip certificate injection
|
||||||
--vm NAME Operate on a single VM only
|
--vm NAME Operate on a single VM only
|
||||||
-y, --yes Skip confirmation prompts
|
-y, --yes Skip confirmation prompts
|
||||||
|
--doctor Check tool versions and environment health
|
||||||
|
--status Check deployment status (shortcut for --phase status)
|
||||||
--cleanup Destroy VMs defined in config
|
--cleanup Destroy VMs defined in config
|
||||||
|
--cleanup --deep Also remove ISOs, seeds, remotes, and local cache
|
||||||
|
--cleanup-all Destroy ALL managed VMs in the resource pool
|
||||||
|
--force-cleanup Destroy VMs even without management marker
|
||||||
--dry-run Preview without executing
|
--dry-run Preview without executing
|
||||||
-q, --quiet Suppress info output
|
-q, --quiet Suppress info output
|
||||||
-h, --help Full help text
|
-h, --help Full help text
|
||||||
-V, --version Show version
|
-V, --version Show version
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### Doctor Check
|
||||||
|
|
||||||
|
Run `--doctor` to verify your environment without a config file:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./incusos-proxmox --doctor
|
||||||
|
```
|
||||||
|
|
||||||
|
This checks: incus client, flasher-tool, operations-center CLI (optional),
|
||||||
|
python3, curl, genisoimage, jq, incusos-seed sibling, and IncusOS CDN
|
||||||
|
availability.
|
||||||
|
|
||||||
|
If a config file is also provided, additionally checks Proxmox connectivity,
|
||||||
|
pool existence, and API token validity:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./incusos-proxmox --doctor examples/lab-cluster.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
### Deep Cleanup
|
||||||
|
|
||||||
|
Use `--deep` with `--cleanup` or `--cleanup-all` to remove all artifacts:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# VMs + ISOs + seeds + incus remotes + local cache
|
||||||
|
./incusos-proxmox --cleanup --deep examples/lab-cluster.yaml
|
||||||
|
|
||||||
|
# Pool-wide: destroy all managed VMs + artifacts
|
||||||
|
./incusos-proxmox --cleanup-all --deep examples/lab-cluster.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
`--deep` additionally:
|
||||||
|
- Deletes seed ISOs from Proxmox storage
|
||||||
|
- Deletes IncusOS ISOs from Proxmox storage
|
||||||
|
- Removes matching `incus remote` entries
|
||||||
|
- Clears the local ISO cache (`~/.cache/incusos-proxmox/iso-cache/`)
|
||||||
|
|
||||||
### Deployment Phases
|
### Deployment Phases
|
||||||
|
|
||||||
| Phase | What it does |
|
| Phase | What it does |
|
||||||
|
|
@ -308,6 +354,7 @@ This script ensures all settings are correct, preventing install failures.
|
||||||
| `upload` | Upload ISO and seed images to Proxmox ISO storage |
|
| `upload` | Upload ISO and seed images to Proxmox ISO storage |
|
||||||
| `create` | Create VMs with IncusOS-correct settings (UEFI, TPM, VirtIO-SCSI, etc.) |
|
| `create` | Create VMs with IncusOS-correct settings (UEFI, TPM, VirtIO-SCSI, etc.) |
|
||||||
| `install` | Start VMs, detect install completion via disk I/O monitoring, detach CD-ROMs, boot from disk |
|
| `install` | Start VMs, detect install completion via disk I/O monitoring, detach CD-ROMs, boot from disk |
|
||||||
|
| `status` | Check deployment status: Proxmox state, IP, port 8443, incus remotes, post-deploy checks |
|
||||||
|
|
||||||
### Config File Format
|
### Config File Format
|
||||||
|
|
||||||
|
|
@ -318,6 +365,7 @@ proxmox:
|
||||||
storage: local-lvm # VM disk storage (default: local-lvm)
|
storage: local-lvm # VM disk storage (default: local-lvm)
|
||||||
iso_storage: local # ISO storage (default: local)
|
iso_storage: local # ISO storage (default: local)
|
||||||
bridge: vmbr0 # Network bridge (default: vmbr0)
|
bridge: vmbr0 # Network bridge (default: vmbr0)
|
||||||
|
pool: IncusLab # Resource pool (optional, for isolation)
|
||||||
method: ssh # ssh (default) or api
|
method: ssh # ssh (default) or api
|
||||||
ssh_user: root # For ssh method
|
ssh_user: root # For ssh method
|
||||||
|
|
||||||
|
|
@ -364,6 +412,54 @@ pveum user token add automation@pve deploy --privsep 0
|
||||||
export PROXMOX_TOKEN_SECRET="<secret-from-above>"
|
export PROXMOX_TOKEN_SECRET="<secret-from-above>"
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### Resource Pool Isolation
|
||||||
|
|
||||||
|
For tighter access control, create a Proxmox resource pool and scope the API
|
||||||
|
token to it instead of `/`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# On Proxmox (one-time setup)
|
||||||
|
pveum pool add IncusLab --comment "IncusOS Lab VMs"
|
||||||
|
pveum pool modify IncusLab --storage local-lvm,local
|
||||||
|
pveum aclmod /pool/IncusLab -user automation@pve -role IncusOSDeploy
|
||||||
|
```
|
||||||
|
|
||||||
|
Then in your config file:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
proxmox:
|
||||||
|
pool: IncusLab
|
||||||
|
```
|
||||||
|
|
||||||
|
When a pool is configured, the script only "sees" VMs in that pool — both for
|
||||||
|
collision detection and cleanup. The VM management marker includes the pool name
|
||||||
|
for traceability.
|
||||||
|
|
||||||
|
### Re-runs, Status, and Reconcile
|
||||||
|
|
||||||
|
**Status check:** Use `--status` to verify deployment health without modifying anything:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./incusos-proxmox --status examples/proxmox-lab.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
Shows per-VM: Proxmox state, install status, IP address, port 8443 reachability,
|
||||||
|
and incus remote connectivity.
|
||||||
|
|
||||||
|
**Re-run safety:** When re-running a full deployment (`--phase all`), the script
|
||||||
|
detects existing VMs and presents an interactive menu:
|
||||||
|
|
||||||
|
1. Run status checks and post-deployment verification
|
||||||
|
2. Continue deployment (skip existing, install incomplete VMs)
|
||||||
|
3. Destroy and redeploy cleanly
|
||||||
|
4. Abort
|
||||||
|
|
||||||
|
With `--yes`, defaults to option 1 (safe — never auto-destroys).
|
||||||
|
|
||||||
|
**Install idempotency:** The install phase checks each VM's state before acting:
|
||||||
|
already-running VMs are skipped, stopped-but-installed VMs are started from disk,
|
||||||
|
and only VMs with install media still attached proceed through installation.
|
||||||
|
|
||||||
### Proxmox VM Settings
|
### Proxmox VM Settings
|
||||||
|
|
||||||
Every VM is created with these IncusOS-specific settings:
|
Every VM is created with these IncusOS-specific settings:
|
||||||
|
|
@ -393,6 +489,60 @@ Every VM is created with these IncusOS-specific settings:
|
||||||
- [`examples/proxmox-lab.yaml`](examples/proxmox-lab.yaml) -- Full 4-VM lab (ops-center + 3-node cluster)
|
- [`examples/proxmox-lab.yaml`](examples/proxmox-lab.yaml) -- Full 4-VM lab (ops-center + 3-node cluster)
|
||||||
- [`examples/proxmox-minimal.yaml`](examples/proxmox-minimal.yaml) -- Single standalone Incus VM
|
- [`examples/proxmox-minimal.yaml`](examples/proxmox-minimal.yaml) -- Single standalone Incus VM
|
||||||
|
|
||||||
|
## lab-test
|
||||||
|
|
||||||
|
Guided lab validation script for IncusOS deployments. Walks through workload
|
||||||
|
testing, cluster formation, and migration tests step by step. Interactive by
|
||||||
|
default — prints what it will do, confirms, executes, and reports pass/fail.
|
||||||
|
|
||||||
|
### Quick Start
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Run all phases (deploy, single-node tests, cluster, workloads, migration)
|
||||||
|
./incusos/lab-test examples/lab-cluster.yaml
|
||||||
|
|
||||||
|
# Deploy and verify only
|
||||||
|
./incusos/lab-test --phase deploy examples/lab-cluster.yaml
|
||||||
|
|
||||||
|
# Single-node workload tests
|
||||||
|
./incusos/lab-test --phase single examples/lab-cluster.yaml
|
||||||
|
|
||||||
|
# Form cluster from deployed nodes
|
||||||
|
./incusos/lab-test --phase cluster examples/lab-cluster.yaml
|
||||||
|
|
||||||
|
# Test migration and evacuation
|
||||||
|
./incusos/lab-test --phase migrate examples/lab-cluster.yaml
|
||||||
|
|
||||||
|
# Full run without prompts
|
||||||
|
./incusos/lab-test --yes examples/lab-cluster.yaml
|
||||||
|
|
||||||
|
# Clean up test instances (keeps VMs)
|
||||||
|
./incusos/lab-test --cleanup examples/lab-cluster.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
### Phases
|
||||||
|
|
||||||
|
| Phase | What it does |
|
||||||
|
|-------|-------------|
|
||||||
|
| `deploy` | Deploy VMs (via incusos-proxmox), verify incus remotes, check connectivity |
|
||||||
|
| `single` | Launch a test container and VM on the first node, exec, cleanup |
|
||||||
|
| `cluster` | Enable clustering on node 1, join remaining nodes, verify membership |
|
||||||
|
| `workload` | Launch instances on the cluster (scheduler-placed + targeted), cleanup |
|
||||||
|
| `migrate` | Stop/move container between nodes, evacuate/restore a node |
|
||||||
|
|
||||||
|
### Options
|
||||||
|
|
||||||
|
```
|
||||||
|
-p, --phase PHASE deploy, single, cluster, workload, migrate, all (default)
|
||||||
|
--skip-deploy Skip deploy/status check (assume VMs already running)
|
||||||
|
--cleanup Remove test instances without destroying VMs
|
||||||
|
-y, --yes Skip confirmation prompts
|
||||||
|
--dry-run Preview actions
|
||||||
|
-q, --quiet Suppress info output
|
||||||
|
-h, --help Full help text
|
||||||
|
-V, --version Show version
|
||||||
|
```
|
||||||
|
|
||||||
## Client Certificate Auto-Detection
|
## Client Certificate Auto-Detection
|
||||||
|
|
||||||
Both scripts automatically look for a client certificate in this order:
|
Both scripts automatically look for a client certificate in this order:
|
||||||
|
|
@ -454,6 +604,8 @@ See the [`examples/`](examples/) directory for sample seed configurations:
|
||||||
- [`network-static.yaml`](examples/network-static.yaml) -- Static network configuration
|
- [`network-static.yaml`](examples/network-static.yaml) -- Static network configuration
|
||||||
- [`proxmox-lab.yaml`](examples/proxmox-lab.yaml) -- Proxmox: 4-VM lab (ops-center + 3-node cluster)
|
- [`proxmox-lab.yaml`](examples/proxmox-lab.yaml) -- Proxmox: 4-VM lab (ops-center + 3-node cluster)
|
||||||
- [`proxmox-minimal.yaml`](examples/proxmox-minimal.yaml) -- Proxmox: single standalone VM
|
- [`proxmox-minimal.yaml`](examples/proxmox-minimal.yaml) -- Proxmox: single standalone VM
|
||||||
|
- [`lab-cluster.yaml`](examples/lab-cluster.yaml) -- Proxmox: 3-node cluster for lab validation
|
||||||
|
- [`lab-oc.yaml`](examples/lab-oc.yaml) -- Proxmox: Operations Center + 3 nodes
|
||||||
|
|
||||||
## Security Notes
|
## Security Notes
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -196,7 +196,23 @@ Using VMID 900+ avoids clashing with existing VMs.
|
||||||
disk, and reboot
|
disk, and reboot
|
||||||
- After reboot you should see the IncusOS login prompt
|
- After reboot you should see the IncusOS login prompt
|
||||||
|
|
||||||
### Step 5: Verify the Deployed VM
|
### Step 5: Check Deployment Status
|
||||||
|
|
||||||
|
After the deploy completes, verify with the built-in status command:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./incusos/incusos-proxmox --status my-test.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
**Verify in the output:**
|
||||||
|
|
||||||
|
- VM `incus-test` exists and is managed
|
||||||
|
- Status is `running`, install state is `installed`
|
||||||
|
- IP address detected
|
||||||
|
- Port 8443 is open
|
||||||
|
- Incus remote status shown (if configured)
|
||||||
|
|
||||||
|
### Step 6: Verify the Deployed VM
|
||||||
|
|
||||||
Once the script reports success with an IP address:
|
Once the script reports success with an IP address:
|
||||||
|
|
||||||
|
|
@ -217,7 +233,26 @@ incus network list incus-test:
|
||||||
ARP-based lookup (MAC from Proxmox VM config → ARP table). If the IP is wrong,
|
ARP-based lookup (MAC from Proxmox VM config → ARP table). If the IP is wrong,
|
||||||
check the Proxmox console — stale ARP entries can be misleading.
|
check the Proxmox console — stale ARP entries can be misleading.
|
||||||
|
|
||||||
### Step 6: Clean Up the Test VM
|
### Step 7: Test Re-run Reconcile
|
||||||
|
|
||||||
|
Re-run the deploy to verify the reconcile menu:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./incusos/incusos-proxmox my-test.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
**Verify:** The script detects the existing VM and shows the interactive menu.
|
||||||
|
Choose option 1 to run status checks without modifying anything.
|
||||||
|
|
||||||
|
Also test with `--yes` to verify it defaults to safe behavior:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./incusos/incusos-proxmox --yes my-test.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
**Verify:** Runs post-deployment checks, does NOT destroy.
|
||||||
|
|
||||||
|
### Step 8: Clean Up the Test VM
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
./incusos/incusos-proxmox --cleanup my-test.yaml
|
./incusos/incusos-proxmox --cleanup my-test.yaml
|
||||||
|
|
@ -234,7 +269,7 @@ Remove the remote too:
|
||||||
incus remote remove incus-test
|
incus remote remove incus-test
|
||||||
```
|
```
|
||||||
|
|
||||||
### Step 7: (Optional) Test API Method
|
### Step 9: (Optional) Test API Method
|
||||||
|
|
||||||
After single-VM success with SSH, switch to API tokens.
|
After single-VM success with SSH, switch to API tokens.
|
||||||
|
|
||||||
|
|
@ -269,7 +304,34 @@ proxmox:
|
||||||
|
|
||||||
Repeat steps 3-6 using the API method.
|
Repeat steps 3-6 using the API method.
|
||||||
|
|
||||||
### Step 8: (Optional) Full Lab Deployment
|
### Step 10: (Optional) Test with Resource Pool
|
||||||
|
|
||||||
|
For API method with pool isolation:
|
||||||
|
|
||||||
|
**On Proxmox:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pveum pool add IncusLab --comment "IncusOS Lab VMs"
|
||||||
|
pveum pool modify IncusLab --storage local-lvm,local
|
||||||
|
pveum aclmod /pool/IncusLab -user automation@pve -role IncusOSDeploy
|
||||||
|
```
|
||||||
|
|
||||||
|
Add to your config:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
proxmox:
|
||||||
|
pool: IncusLab
|
||||||
|
```
|
||||||
|
|
||||||
|
Deploy and verify:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./incusos/incusos-proxmox --dry-run my-test.yaml # Should show pool in plan
|
||||||
|
./incusos/incusos-proxmox my-test.yaml # VM created in pool
|
||||||
|
./incusos/incusos-proxmox --status my-test.yaml # Status scoped to pool
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 11: (Optional) Full Lab Deployment
|
||||||
|
|
||||||
Once single-VM tests pass, deploy the full 4-VM lab:
|
Once single-VM tests pass, deploy the full 4-VM lab:
|
||||||
|
|
||||||
|
|
@ -288,6 +350,144 @@ This deploys 1 Operations Center + 3 Incus nodes. After deployment:
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## Lab Validation with lab-test
|
||||||
|
|
||||||
|
The `lab-test` script automates the end-to-end validation of deployed IncusOS
|
||||||
|
VMs: single-node workloads, cluster formation, migration, and evacuation.
|
||||||
|
|
||||||
|
### Quick Start: Cluster Testing
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. Check environment
|
||||||
|
./incusos/incusos-proxmox --doctor
|
||||||
|
|
||||||
|
# 2. Deploy 3 nodes
|
||||||
|
./incusos/incusos-proxmox --yes examples/lab-cluster.yaml
|
||||||
|
|
||||||
|
# 3. Add incus remotes for each node (use IPs from --status)
|
||||||
|
./incusos/incusos-proxmox --status examples/lab-cluster.yaml
|
||||||
|
incus remote add incus-lab-01 <IP1> --accept-certificate
|
||||||
|
incus remote add incus-lab-02 <IP2> --accept-certificate
|
||||||
|
incus remote add incus-lab-03 <IP3> --accept-certificate
|
||||||
|
|
||||||
|
# 4. Run full lab validation
|
||||||
|
./incusos/lab-test --yes examples/lab-cluster.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
### Phase-by-Phase Testing
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Single-node workloads only (container + VM launch/exec)
|
||||||
|
./incusos/lab-test --phase single --skip-deploy examples/lab-cluster.yaml
|
||||||
|
|
||||||
|
# Cluster formation only
|
||||||
|
./incusos/lab-test --phase cluster --skip-deploy examples/lab-cluster.yaml
|
||||||
|
|
||||||
|
# Cluster workloads (scheduler placement + targeted placement)
|
||||||
|
./incusos/lab-test --phase workload --skip-deploy examples/lab-cluster.yaml
|
||||||
|
|
||||||
|
# Migration and evacuation tests
|
||||||
|
./incusos/lab-test --phase migrate --skip-deploy examples/lab-cluster.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
### Manual Cluster Formation (Alternative)
|
||||||
|
|
||||||
|
If you prefer to form the cluster manually instead of using `lab-test`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Step 1: Enable clustering on first node
|
||||||
|
incus cluster enable incus-lab-01: incus-lab-01
|
||||||
|
|
||||||
|
# Step 2: Generate join token for second node
|
||||||
|
incus cluster add incus-lab-01: incus-lab-02
|
||||||
|
|
||||||
|
# Step 3: Join second node
|
||||||
|
incus cluster join incus-lab-01: incus-lab-02:
|
||||||
|
|
||||||
|
# Step 4: Repeat for third node
|
||||||
|
incus cluster add incus-lab-01: incus-lab-03
|
||||||
|
incus cluster join incus-lab-01: incus-lab-03:
|
||||||
|
|
||||||
|
# Step 5: Verify
|
||||||
|
incus cluster list incus-lab-01:
|
||||||
|
```
|
||||||
|
|
||||||
|
**Key points:**
|
||||||
|
- `apply_defaults: true` on all nodes ensures matching storage pool and
|
||||||
|
network bridge configurations (required for migration)
|
||||||
|
- After joining, manage the cluster through the init node's remote
|
||||||
|
- VM migration requires matching storage pool names across nodes
|
||||||
|
|
||||||
|
### Cleanup
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Remove test instances only (keeps VMs and cluster)
|
||||||
|
./incusos/lab-test --cleanup examples/lab-cluster.yaml
|
||||||
|
|
||||||
|
# Destroy VMs only
|
||||||
|
./incusos/incusos-proxmox --cleanup examples/lab-cluster.yaml
|
||||||
|
|
||||||
|
# Full cleanup: VMs + ISOs + seeds + remotes + cache
|
||||||
|
./incusos/incusos-proxmox --cleanup --deep --yes examples/lab-cluster.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Operations Center Lab
|
||||||
|
|
||||||
|
### Deploy
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./incusos/incusos-proxmox --yes examples/lab-oc.yaml
|
||||||
|
./incusos/incusos-proxmox --status examples/lab-oc.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
### Access the Web UI
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Export client certificate for browser
|
||||||
|
openssl pkcs12 -export \
|
||||||
|
-inkey ~/.config/incus/client.key \
|
||||||
|
-in ~/.config/incus/client.crt \
|
||||||
|
-out client.pfx
|
||||||
|
|
||||||
|
# Import client.pfx into Firefox, navigate to https://<OC_IP>:8443
|
||||||
|
```
|
||||||
|
|
||||||
|
### OC CLI Setup
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Copy incus client certs for OC CLI
|
||||||
|
mkdir -p ~/.config/operations-center
|
||||||
|
cp ~/.config/incus/client.crt ~/.config/operations-center/
|
||||||
|
cp ~/.config/incus/client.key ~/.config/operations-center/
|
||||||
|
|
||||||
|
# Add OC remote
|
||||||
|
operations-center remote add lab-oc https://<OC_IP>:8443
|
||||||
|
operations-center remote switch lab-oc
|
||||||
|
|
||||||
|
# Check for provisioning updates
|
||||||
|
operations-center provisioning update list
|
||||||
|
```
|
||||||
|
|
||||||
|
### Provisioning Workflow
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Create a provisioning token
|
||||||
|
operations-center provisioning token add --description "lab nodes" --uses 10
|
||||||
|
|
||||||
|
# Get provisioned ISO
|
||||||
|
operations-center provisioning token get-image <token> ~/Downloads/IncusOS-provisioned.iso
|
||||||
|
|
||||||
|
# Redeploy nodes with provisioned ISO, or check if manual adoption works
|
||||||
|
```
|
||||||
|
|
||||||
|
**Note:** Operations Center is under active development (v0.2.2+). Commands and
|
||||||
|
APIs may change between versions. The `--doctor` command reports whether the OC
|
||||||
|
CLI is installed.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## Troubleshooting
|
## Troubleshooting
|
||||||
|
|
||||||
| Symptom | Likely Cause | Fix |
|
| Symptom | Likely Cause | Fix |
|
||||||
|
|
@ -304,3 +504,6 @@ This deploys 1 Operations Center + 3 Incus nodes. After deployment:
|
||||||
| "field server not found" YAML error | Wrong preseed structure | Use `preseed.certificates[]` not `preseed.server.certificates[]` |
|
| "field server not found" YAML error | Wrong preseed structure | Use `preseed.certificates[]` not `preseed.server.certificates[]` |
|
||||||
| Cert not auto-detected | No Incus keypair | Run `incus remote list` to generate one |
|
| Cert not auto-detected | No Incus keypair | Run `incus remote list` to generate one |
|
||||||
| `genisoimage` not found | genisoimage not installed | Install `genisoimage` package |
|
| `genisoimage` not found | genisoimage not installed | Install `genisoimage` package |
|
||||||
|
| Pool not found | Resource pool not created | Run `pveum pool add <name>` on Proxmox |
|
||||||
|
| Re-run destroys VMs | Used option 3 in reconcile | Use option 1 (checks) or option 2 (continue) instead |
|
||||||
|
| `--status` shows "not found" | VM was cleaned up | Deploy first, then check status |
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,44 @@
|
||||||
|
# lab-cluster.yaml - 3-node IncusOS cluster for lab testing
|
||||||
|
#
|
||||||
|
# Deploys 3 standalone Incus nodes that can be clustered after deployment.
|
||||||
|
# All nodes get apply_defaults: true so they have matching storage pool
|
||||||
|
# and network bridge configurations.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# incusos-proxmox --dry-run examples/lab-cluster.yaml # Preview
|
||||||
|
# incusos-proxmox --yes examples/lab-cluster.yaml # Deploy
|
||||||
|
# incusos-proxmox --status examples/lab-cluster.yaml # Check status
|
||||||
|
# incusos-proxmox --cleanup examples/lab-cluster.yaml # Tear down VMs
|
||||||
|
# incusos-proxmox --cleanup --deep examples/lab-cluster.yaml # Full cleanup
|
||||||
|
#
|
||||||
|
# After deployment, form a cluster with lab-test:
|
||||||
|
# lab-test --phase cluster examples/lab-cluster.yaml
|
||||||
|
|
||||||
|
proxmox:
|
||||||
|
host: 192.168.1.29
|
||||||
|
method: api
|
||||||
|
api_token_id: automation@pve!deploy
|
||||||
|
storage: local-zfs
|
||||||
|
pool: IncusLab
|
||||||
|
|
||||||
|
defaults:
|
||||||
|
cores: 2
|
||||||
|
memory: 4096
|
||||||
|
disk: 50
|
||||||
|
start_vmid: 900
|
||||||
|
|
||||||
|
vms:
|
||||||
|
- name: incus-lab-01
|
||||||
|
app: incus
|
||||||
|
apply_defaults: true
|
||||||
|
cores: 4
|
||||||
|
memory: 8192
|
||||||
|
disk: 64
|
||||||
|
|
||||||
|
- name: incus-lab-02
|
||||||
|
app: incus
|
||||||
|
apply_defaults: true
|
||||||
|
|
||||||
|
- name: incus-lab-03
|
||||||
|
app: incus
|
||||||
|
apply_defaults: true
|
||||||
|
|
@ -0,0 +1,50 @@
|
||||||
|
# lab-oc.yaml - Operations Center + 3-node lab
|
||||||
|
#
|
||||||
|
# Deploys 1 Operations Center instance and 3 Incus nodes.
|
||||||
|
# Uses VMID range 910+ to avoid collisions with lab-cluster.yaml (900+).
|
||||||
|
#
|
||||||
|
# Workflow:
|
||||||
|
# 1. Deploy with incusos-proxmox
|
||||||
|
# 2. Access OC web UI (import client.pfx into browser)
|
||||||
|
# 3. Create provisioning token in OC
|
||||||
|
# 4. Either redeploy nodes with OC-provisioned ISO or manually adopt
|
||||||
|
# 5. Form cluster via OC CLI or web UI
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# incusos-proxmox --dry-run examples/lab-oc.yaml
|
||||||
|
# incusos-proxmox --yes examples/lab-oc.yaml
|
||||||
|
# incusos-proxmox --status examples/lab-oc.yaml
|
||||||
|
# incusos-proxmox --cleanup --deep examples/lab-oc.yaml
|
||||||
|
|
||||||
|
proxmox:
|
||||||
|
host: 192.168.1.29
|
||||||
|
method: api
|
||||||
|
api_token_id: automation@pve!deploy
|
||||||
|
storage: local-zfs
|
||||||
|
pool: IncusLab
|
||||||
|
|
||||||
|
defaults:
|
||||||
|
cores: 2
|
||||||
|
memory: 4096
|
||||||
|
disk: 50
|
||||||
|
start_vmid: 910
|
||||||
|
|
||||||
|
vms:
|
||||||
|
- name: lab-oc
|
||||||
|
app: operations-center
|
||||||
|
apply_defaults: true
|
||||||
|
|
||||||
|
- name: lab-node-01
|
||||||
|
app: incus
|
||||||
|
apply_defaults: true
|
||||||
|
cores: 4
|
||||||
|
memory: 8192
|
||||||
|
disk: 64
|
||||||
|
|
||||||
|
- name: lab-node-02
|
||||||
|
app: incus
|
||||||
|
apply_defaults: true
|
||||||
|
|
||||||
|
- name: lab-node-03
|
||||||
|
app: incus
|
||||||
|
apply_defaults: true
|
||||||
|
|
@ -3,15 +3,27 @@ proxmox:
|
||||||
method: api
|
method: api
|
||||||
api_token_id: automation@pve!deploy
|
api_token_id: automation@pve!deploy
|
||||||
storage: local-zfs # VM disk storage (default: local-lvm)
|
storage: local-zfs # VM disk storage (default: local-lvm)
|
||||||
|
pool: IncusLab
|
||||||
|
|
||||||
defaults:
|
defaults:
|
||||||
|
cores: 2
|
||||||
|
memory: 4096 # Minimum for IncusOS
|
||||||
|
disk: 50 # Minimum for IncusOS
|
||||||
start_vmid: 900 # High range to avoid collisions
|
start_vmid: 900 # High range to avoid collisions
|
||||||
|
|
||||||
vms:
|
vms:
|
||||||
- name: incus-test
|
- name: incus-lab-01
|
||||||
app: incus
|
app: incus
|
||||||
apply_defaults: true
|
apply_defaults: true
|
||||||
cores: 4
|
cores: 4
|
||||||
memory: 8192
|
memory: 8192
|
||||||
disk: 64
|
disk: 64
|
||||||
|
|
||||||
|
- name: incus-lab-02
|
||||||
|
app: incus
|
||||||
|
apply_defaults: true
|
||||||
|
|
||||||
|
- name: ops-center
|
||||||
|
app: operations-center
|
||||||
|
apply_defaults: true
|
||||||
|
|
||||||
|
|
@ -15,6 +15,7 @@ proxmox:
|
||||||
storage: local-lvm # VM disk storage
|
storage: local-lvm # VM disk storage
|
||||||
iso_storage: local # ISO/image storage
|
iso_storage: local # ISO/image storage
|
||||||
bridge: vmbr0 # Network bridge
|
bridge: vmbr0 # Network bridge
|
||||||
|
# pool: IncusLab # Resource pool (optional, for isolation)
|
||||||
method: ssh # ssh (default) or api
|
method: ssh # ssh (default) or api
|
||||||
ssh_user: root # SSH user for ssh method
|
ssh_user: root # SSH user for ssh method
|
||||||
# api_token_id: automation@pve!deploy # For api method
|
# api_token_id: automation@pve!deploy # For api method
|
||||||
|
|
|
||||||
|
|
@ -8,6 +8,7 @@
|
||||||
|
|
||||||
proxmox:
|
proxmox:
|
||||||
host: 192.168.1.10 # Change to your Proxmox host
|
host: 192.168.1.10 # Change to your Proxmox host
|
||||||
|
# pool: IncusLab # Resource pool (optional, for isolation)
|
||||||
# All other settings use defaults:
|
# All other settings use defaults:
|
||||||
# node: pve, storage: local-lvm, iso_storage: local,
|
# node: pve, storage: local-lvm, iso_storage: local,
|
||||||
# bridge: vmbr0, method: ssh, ssh_user: root
|
# bridge: vmbr0, method: ssh, ssh_user: root
|
||||||
|
|
|
||||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
Loading…
Reference in New Issue