Add lab validation tooling: doctor check, deep cleanup, lab-test script
Implements the first wave of lab validation infrastructure: incusos-proxmox enhancements: - --doctor flag: standalone environment health check (tool versions, IncusOS CDN availability, Proxmox connectivity). Works with or without a config file. - --cleanup --deep: extended cleanup that also removes seed ISOs, IncusOS ISOs from Proxmox storage, matching incus remotes, and the local ISO cache (~/.cache/incusos-proxmox/iso-cache/). - --cleanup-all: pool-wide cleanup that destroys ALL VMs with the [incusos-lab:managed] marker in the configured resource pool. Requires a pool to be set in config for safety scoping. New script -- lab-test: - Guided lab validation with 5 phases: deploy & verify, single-node workloads (container + VM lifecycle), cluster formation (enable + join via incus remotes), cluster workloads (scheduler-placed + targeted), and migration & evacuation (stop/move + evacuate/restore). - Interactive by default with --yes for unattended runs. - Reports PASS/FAIL/SKIP per test with a summary. - Supports --dry-run, --skip-deploy, --cleanup, per-phase execution. - Reads the same YAML config format as incusos-proxmox. New example configs: - lab-cluster.yaml: 3-node Incus cluster (VMID 900+) for testing clustering, workloads, and migration. - lab-oc.yaml: Operations Center + 3 Incus nodes (VMID 910+) for testing OC-managed provisioning and cluster formation. Documentation: - CLAUDE.md: added clustering workflow (incus cluster enable/add/join via remotes), lab-test overview, Operations Center CLI notes, doctor and cleanup feature descriptions. - README.md: added lab-test section with usage/phases/options, doctor check docs, deep cleanup docs, new example files in listing. - TESTING.md: added full lab validation guide (quick start, phase-by- phase testing, manual cluster formation alternative, OC lab workflow with CLI setup and provisioning steps), cleanup instructions. Not yet tested against live infrastructure -- next step is to deploy lab-cluster.yaml, add incus remotes, and run lab-test end-to-end. Known areas to validate: cluster join syntax with remotes, VM nested virt support, evacuation behavior, apply_defaults consistency across cluster members. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
a0a4b48c73
commit
2aca6fb33b
78
CLAUDE.md
78
CLAUDE.md
|
|
@ -18,7 +18,8 @@ incu-contrib/
|
|||
│ ├── incusos-iso # ISO/IMG builder (wraps flasher-tool)
|
||||
│ ├── incusos-seed # Seed archive generator
|
||||
│ ├── incusos-proxmox # Declarative Proxmox VM deployment
|
||||
│ ├── TESTING.md # Testing guide for incusos-proxmox
|
||||
│ ├── lab-test # Guided lab validation (workloads, clustering, migration)
|
||||
│ ├── TESTING.md # Testing guide for incusos-proxmox and lab-test
|
||||
│ └── examples/ # Example seed + Proxmox YAML files
|
||||
└── notes/ # Research notes and reference material
|
||||
```
|
||||
|
|
@ -116,6 +117,81 @@ incu-contrib/
|
|||
IncusOS sits at "please remove installation media" and waits indefinitely.
|
||||
It does NOT halt automatically. `force_reboot` triggers a guest-level reboot
|
||||
(note: this does NOT reset QEMU VM uptime -- only a Proxmox stop/start does).
|
||||
- **Resource pool isolation**: the optional `proxmox.pool` config field scopes
|
||||
all VM operations to a Proxmox resource pool. When set, the script only
|
||||
"sees" VMs in that pool (for collision detection and cleanup), and the API
|
||||
token ACL can be scoped to `/pool/<name>` instead of `/`. Setup:
|
||||
```
|
||||
pveum pool add IncusLab --comment "IncusOS Lab VMs"
|
||||
pveum pool modify IncusLab --storage local-lvm,local
|
||||
pveum aclmod /pool/IncusLab -user automation@pve -role IncusOSDeploy
|
||||
```
|
||||
- **`--status` command**: `incusos-proxmox --status config.yaml` shows per-VM
|
||||
deployment status (Proxmox state, install status, IP, port 8443, incus
|
||||
remote). Also runs post-deployment checks (Incus connectivity, Operations
|
||||
Center URL).
|
||||
- **Reconcile on re-runs**: when `--phase all` detects existing VMs from
|
||||
config, an interactive menu offers: (1) run status checks, (2) continue
|
||||
install for incomplete VMs, (3) destroy and redeploy, (4) abort. With
|
||||
`--yes`, defaults to option 1 (safe -- never auto-destroys).
|
||||
- **Install idempotency**: `phase_install` checks each VM's state before
|
||||
acting -- already-running VMs are skipped, stopped-but-installed VMs are
|
||||
started from disk, only VMs with install media proceed through installation.
|
||||
|
||||
### Incus clustering via remotes
|
||||
|
||||
- **Cluster formation** is done entirely through the `incus` CLI using remotes.
|
||||
No SSH to the IncusOS nodes is needed (IncusOS is immutable, no shell access).
|
||||
- **Workflow:**
|
||||
```bash
|
||||
# Enable clustering on first node
|
||||
incus cluster enable <remote1>: <member-name>
|
||||
|
||||
# Generate join token (on cluster, for new member name)
|
||||
incus cluster add <remote1>: <new-member-name>
|
||||
|
||||
# Join second node to cluster
|
||||
incus cluster join <remote1>: <remote2>:
|
||||
|
||||
# Verify
|
||||
incus cluster list <remote1>:
|
||||
```
|
||||
- **`apply_defaults: true`** must be set on all nodes that will be clustered --
|
||||
this ensures matching storage pool (`local`) and network bridge names, which
|
||||
are required for migration.
|
||||
- After joining, the cluster is managed through the init node's remote. The
|
||||
individual node remotes still work for node-specific operations.
|
||||
- **`lab-test`** automates cluster formation, workload testing, and migration.
|
||||
|
||||
### Lab validation (lab-test)
|
||||
|
||||
- **`lab-test`** reads the same YAML config as `incusos-proxmox` and operates
|
||||
on the VM names defined there (expects `incus` remotes to exist).
|
||||
- **Phases:** deploy, single (workloads), cluster, workload (on cluster),
|
||||
migrate (stop/move + evacuate/restore).
|
||||
- **Test instances** use names starting with `test-` for easy cleanup.
|
||||
- The script reports PASS/FAIL/SKIP for each test and prints a summary.
|
||||
|
||||
### Operations Center
|
||||
|
||||
- **CLI binary:** `operations-center` (installed from GitHub releases or
|
||||
built from source at `github.com/FuturFusion/operations-center`).
|
||||
- **Config directory:** `~/.config/operations-center/` (uses same cert format
|
||||
as Incus: copy `client.crt` and `client.key` from `~/.config/incus/`).
|
||||
- **Provisioning workflow:** create token → get provisioned ISO → deploy nodes
|
||||
with that ISO → nodes auto-register with OC → form cluster via OC.
|
||||
- **OC is under active development** (v0.2.2+). Commands and APIs may change.
|
||||
The `--doctor` command on `incusos-proxmox` reports whether the CLI is
|
||||
installed.
|
||||
|
||||
### incusos-proxmox doctor and cleanup
|
||||
|
||||
- **`--doctor`**: standalone environment check. No config file required.
|
||||
Checks tool versions, IncusOS CDN, and optionally Proxmox connectivity.
|
||||
- **`--cleanup --deep`**: destroys VMs + deletes ISOs/seeds from Proxmox +
|
||||
removes incus remotes + clears local ISO cache.
|
||||
- **`--cleanup-all`**: pool-wide cleanup. Destroys all VMs with the
|
||||
`[incusos-lab:managed]` marker in the configured resource pool.
|
||||
|
||||
## Coding conventions for scripts
|
||||
|
||||
|
|
|
|||
|
|
@ -9,6 +9,7 @@ Scripts for building IncusOS installation media and seed configurations.
|
|||
| [`incusos-iso`](#incusos-iso) | Build customized IncusOS ISO/IMG images using the flasher-tool |
|
||||
| [`incusos-seed`](#incusos-seed) | Generate seed archives (tar, FAT, or ISO 9660) for installation customization |
|
||||
| [`incusos-proxmox`](#incusos-proxmox) | Declarative IncusOS deployment on Proxmox VE |
|
||||
| [`lab-test`](#lab-test) | Guided lab validation (workloads, clustering, migration) |
|
||||
|
||||
All scripts support automatic client certificate injection from the local Incus
|
||||
installation, so the freshly installed system is immediately manageable.
|
||||
|
|
@ -274,6 +275,9 @@ This script ensures all settings are correct, preventing install failures.
|
|||
# Deploy everything
|
||||
./incusos-proxmox examples/proxmox-lab.yaml
|
||||
|
||||
# Check deployment status
|
||||
./incusos-proxmox --status examples/proxmox-lab.yaml
|
||||
|
||||
# Deploy a single VM from the config
|
||||
./incusos-proxmox --vm ops-center examples/proxmox-lab.yaml
|
||||
|
||||
|
|
@ -284,7 +288,7 @@ This script ensures all settings are correct, preventing install failures.
|
|||
### Options Reference
|
||||
|
||||
```
|
||||
-p, --phase PHASE validate, prepare, upload, create, install, all (default)
|
||||
-p, --phase PHASE validate, prepare, upload, create, install, status, all (default)
|
||||
-m, --method METHOD Override connection method: ssh (default), api
|
||||
-H, --host HOST Override Proxmox host from config
|
||||
-i, --iso FILE Use local ISO (skip CDN download)
|
||||
|
|
@ -292,13 +296,55 @@ This script ensures all settings are correct, preventing install failures.
|
|||
--no-cert Skip certificate injection
|
||||
--vm NAME Operate on a single VM only
|
||||
-y, --yes Skip confirmation prompts
|
||||
--doctor Check tool versions and environment health
|
||||
--status Check deployment status (shortcut for --phase status)
|
||||
--cleanup Destroy VMs defined in config
|
||||
--cleanup --deep Also remove ISOs, seeds, remotes, and local cache
|
||||
--cleanup-all Destroy ALL managed VMs in the resource pool
|
||||
--force-cleanup Destroy VMs even without management marker
|
||||
--dry-run Preview without executing
|
||||
-q, --quiet Suppress info output
|
||||
-h, --help Full help text
|
||||
-V, --version Show version
|
||||
```
|
||||
|
||||
### Doctor Check
|
||||
|
||||
Run `--doctor` to verify your environment without a config file:
|
||||
|
||||
```bash
|
||||
./incusos-proxmox --doctor
|
||||
```
|
||||
|
||||
This checks: incus client, flasher-tool, operations-center CLI (optional),
|
||||
python3, curl, genisoimage, jq, incusos-seed sibling, and IncusOS CDN
|
||||
availability.
|
||||
|
||||
If a config file is also provided, additionally checks Proxmox connectivity,
|
||||
pool existence, and API token validity:
|
||||
|
||||
```bash
|
||||
./incusos-proxmox --doctor examples/lab-cluster.yaml
|
||||
```
|
||||
|
||||
### Deep Cleanup
|
||||
|
||||
Use `--deep` with `--cleanup` or `--cleanup-all` to remove all artifacts:
|
||||
|
||||
```bash
|
||||
# VMs + ISOs + seeds + incus remotes + local cache
|
||||
./incusos-proxmox --cleanup --deep examples/lab-cluster.yaml
|
||||
|
||||
# Pool-wide: destroy all managed VMs + artifacts
|
||||
./incusos-proxmox --cleanup-all --deep examples/lab-cluster.yaml
|
||||
```
|
||||
|
||||
`--deep` additionally:
|
||||
- Deletes seed ISOs from Proxmox storage
|
||||
- Deletes IncusOS ISOs from Proxmox storage
|
||||
- Removes matching `incus remote` entries
|
||||
- Clears the local ISO cache (`~/.cache/incusos-proxmox/iso-cache/`)
|
||||
|
||||
### Deployment Phases
|
||||
|
||||
| Phase | What it does |
|
||||
|
|
@ -308,6 +354,7 @@ This script ensures all settings are correct, preventing install failures.
|
|||
| `upload` | Upload ISO and seed images to Proxmox ISO storage |
|
||||
| `create` | Create VMs with IncusOS-correct settings (UEFI, TPM, VirtIO-SCSI, etc.) |
|
||||
| `install` | Start VMs, detect install completion via disk I/O monitoring, detach CD-ROMs, boot from disk |
|
||||
| `status` | Check deployment status: Proxmox state, IP, port 8443, incus remotes, post-deploy checks |
|
||||
|
||||
### Config File Format
|
||||
|
||||
|
|
@ -318,6 +365,7 @@ proxmox:
|
|||
storage: local-lvm # VM disk storage (default: local-lvm)
|
||||
iso_storage: local # ISO storage (default: local)
|
||||
bridge: vmbr0 # Network bridge (default: vmbr0)
|
||||
pool: IncusLab # Resource pool (optional, for isolation)
|
||||
method: ssh # ssh (default) or api
|
||||
ssh_user: root # For ssh method
|
||||
|
||||
|
|
@ -364,6 +412,54 @@ pveum user token add automation@pve deploy --privsep 0
|
|||
export PROXMOX_TOKEN_SECRET="<secret-from-above>"
|
||||
```
|
||||
|
||||
### Resource Pool Isolation
|
||||
|
||||
For tighter access control, create a Proxmox resource pool and scope the API
|
||||
token to it instead of `/`:
|
||||
|
||||
```bash
|
||||
# On Proxmox (one-time setup)
|
||||
pveum pool add IncusLab --comment "IncusOS Lab VMs"
|
||||
pveum pool modify IncusLab --storage local-lvm,local
|
||||
pveum aclmod /pool/IncusLab -user automation@pve -role IncusOSDeploy
|
||||
```
|
||||
|
||||
Then in your config file:
|
||||
|
||||
```yaml
|
||||
proxmox:
|
||||
pool: IncusLab
|
||||
```
|
||||
|
||||
When a pool is configured, the script only "sees" VMs in that pool — both for
|
||||
collision detection and cleanup. The VM management marker includes the pool name
|
||||
for traceability.
|
||||
|
||||
### Re-runs, Status, and Reconcile
|
||||
|
||||
**Status check:** Use `--status` to verify deployment health without modifying anything:
|
||||
|
||||
```bash
|
||||
./incusos-proxmox --status examples/proxmox-lab.yaml
|
||||
```
|
||||
|
||||
Shows per-VM: Proxmox state, install status, IP address, port 8443 reachability,
|
||||
and incus remote connectivity.
|
||||
|
||||
**Re-run safety:** When re-running a full deployment (`--phase all`), the script
|
||||
detects existing VMs and presents an interactive menu:
|
||||
|
||||
1. Run status checks and post-deployment verification
|
||||
2. Continue deployment (skip existing, install incomplete VMs)
|
||||
3. Destroy and redeploy cleanly
|
||||
4. Abort
|
||||
|
||||
With `--yes`, defaults to option 1 (safe — never auto-destroys).
|
||||
|
||||
**Install idempotency:** The install phase checks each VM's state before acting:
|
||||
already-running VMs are skipped, stopped-but-installed VMs are started from disk,
|
||||
and only VMs with install media still attached proceed through installation.
|
||||
|
||||
### Proxmox VM Settings
|
||||
|
||||
Every VM is created with these IncusOS-specific settings:
|
||||
|
|
@ -393,6 +489,60 @@ Every VM is created with these IncusOS-specific settings:
|
|||
- [`examples/proxmox-lab.yaml`](examples/proxmox-lab.yaml) -- Full 4-VM lab (ops-center + 3-node cluster)
|
||||
- [`examples/proxmox-minimal.yaml`](examples/proxmox-minimal.yaml) -- Single standalone Incus VM
|
||||
|
||||
## lab-test
|
||||
|
||||
Guided lab validation script for IncusOS deployments. Walks through workload
|
||||
testing, cluster formation, and migration tests step by step. Interactive by
|
||||
default — prints what it will do, confirms, executes, and reports pass/fail.
|
||||
|
||||
### Quick Start
|
||||
|
||||
```bash
|
||||
# Run all phases (deploy, single-node tests, cluster, workloads, migration)
|
||||
./incusos/lab-test examples/lab-cluster.yaml
|
||||
|
||||
# Deploy and verify only
|
||||
./incusos/lab-test --phase deploy examples/lab-cluster.yaml
|
||||
|
||||
# Single-node workload tests
|
||||
./incusos/lab-test --phase single examples/lab-cluster.yaml
|
||||
|
||||
# Form cluster from deployed nodes
|
||||
./incusos/lab-test --phase cluster examples/lab-cluster.yaml
|
||||
|
||||
# Test migration and evacuation
|
||||
./incusos/lab-test --phase migrate examples/lab-cluster.yaml
|
||||
|
||||
# Full run without prompts
|
||||
./incusos/lab-test --yes examples/lab-cluster.yaml
|
||||
|
||||
# Clean up test instances (keeps VMs)
|
||||
./incusos/lab-test --cleanup examples/lab-cluster.yaml
|
||||
```
|
||||
|
||||
### Phases
|
||||
|
||||
| Phase | What it does |
|
||||
|-------|-------------|
|
||||
| `deploy` | Deploy VMs (via incusos-proxmox), verify incus remotes, check connectivity |
|
||||
| `single` | Launch a test container and VM on the first node, exec, cleanup |
|
||||
| `cluster` | Enable clustering on node 1, join remaining nodes, verify membership |
|
||||
| `workload` | Launch instances on the cluster (scheduler-placed + targeted), cleanup |
|
||||
| `migrate` | Stop/move container between nodes, evacuate/restore a node |
|
||||
|
||||
### Options
|
||||
|
||||
```
|
||||
-p, --phase PHASE deploy, single, cluster, workload, migrate, all (default)
|
||||
--skip-deploy Skip deploy/status check (assume VMs already running)
|
||||
--cleanup Remove test instances without destroying VMs
|
||||
-y, --yes Skip confirmation prompts
|
||||
--dry-run Preview actions
|
||||
-q, --quiet Suppress info output
|
||||
-h, --help Full help text
|
||||
-V, --version Show version
|
||||
```
|
||||
|
||||
## Client Certificate Auto-Detection
|
||||
|
||||
Both scripts automatically look for a client certificate in this order:
|
||||
|
|
@ -454,6 +604,8 @@ See the [`examples/`](examples/) directory for sample seed configurations:
|
|||
- [`network-static.yaml`](examples/network-static.yaml) -- Static network configuration
|
||||
- [`proxmox-lab.yaml`](examples/proxmox-lab.yaml) -- Proxmox: 4-VM lab (ops-center + 3-node cluster)
|
||||
- [`proxmox-minimal.yaml`](examples/proxmox-minimal.yaml) -- Proxmox: single standalone VM
|
||||
- [`lab-cluster.yaml`](examples/lab-cluster.yaml) -- Proxmox: 3-node cluster for lab validation
|
||||
- [`lab-oc.yaml`](examples/lab-oc.yaml) -- Proxmox: Operations Center + 3 nodes
|
||||
|
||||
## Security Notes
|
||||
|
||||
|
|
|
|||
|
|
@ -196,7 +196,23 @@ Using VMID 900+ avoids clashing with existing VMs.
|
|||
disk, and reboot
|
||||
- After reboot you should see the IncusOS login prompt
|
||||
|
||||
### Step 5: Verify the Deployed VM
|
||||
### Step 5: Check Deployment Status
|
||||
|
||||
After the deploy completes, verify with the built-in status command:
|
||||
|
||||
```bash
|
||||
./incusos/incusos-proxmox --status my-test.yaml
|
||||
```
|
||||
|
||||
**Verify in the output:**
|
||||
|
||||
- VM `incus-test` exists and is managed
|
||||
- Status is `running`, install state is `installed`
|
||||
- IP address detected
|
||||
- Port 8443 is open
|
||||
- Incus remote status shown (if configured)
|
||||
|
||||
### Step 6: Verify the Deployed VM
|
||||
|
||||
Once the script reports success with an IP address:
|
||||
|
||||
|
|
@ -217,7 +233,26 @@ incus network list incus-test:
|
|||
ARP-based lookup (MAC from Proxmox VM config → ARP table). If the IP is wrong,
|
||||
check the Proxmox console — stale ARP entries can be misleading.
|
||||
|
||||
### Step 6: Clean Up the Test VM
|
||||
### Step 7: Test Re-run Reconcile
|
||||
|
||||
Re-run the deploy to verify the reconcile menu:
|
||||
|
||||
```bash
|
||||
./incusos/incusos-proxmox my-test.yaml
|
||||
```
|
||||
|
||||
**Verify:** The script detects the existing VM and shows the interactive menu.
|
||||
Choose option 1 to run status checks without modifying anything.
|
||||
|
||||
Also test with `--yes` to verify it defaults to safe behavior:
|
||||
|
||||
```bash
|
||||
./incusos/incusos-proxmox --yes my-test.yaml
|
||||
```
|
||||
|
||||
**Verify:** Runs post-deployment checks, does NOT destroy.
|
||||
|
||||
### Step 8: Clean Up the Test VM
|
||||
|
||||
```bash
|
||||
./incusos/incusos-proxmox --cleanup my-test.yaml
|
||||
|
|
@ -234,7 +269,7 @@ Remove the remote too:
|
|||
incus remote remove incus-test
|
||||
```
|
||||
|
||||
### Step 7: (Optional) Test API Method
|
||||
### Step 9: (Optional) Test API Method
|
||||
|
||||
After single-VM success with SSH, switch to API tokens.
|
||||
|
||||
|
|
@ -269,7 +304,34 @@ proxmox:
|
|||
|
||||
Repeat steps 3-6 using the API method.
|
||||
|
||||
### Step 8: (Optional) Full Lab Deployment
|
||||
### Step 10: (Optional) Test with Resource Pool
|
||||
|
||||
For API method with pool isolation:
|
||||
|
||||
**On Proxmox:**
|
||||
|
||||
```bash
|
||||
pveum pool add IncusLab --comment "IncusOS Lab VMs"
|
||||
pveum pool modify IncusLab --storage local-lvm,local
|
||||
pveum aclmod /pool/IncusLab -user automation@pve -role IncusOSDeploy
|
||||
```
|
||||
|
||||
Add to your config:
|
||||
|
||||
```yaml
|
||||
proxmox:
|
||||
pool: IncusLab
|
||||
```
|
||||
|
||||
Deploy and verify:
|
||||
|
||||
```bash
|
||||
./incusos/incusos-proxmox --dry-run my-test.yaml # Should show pool in plan
|
||||
./incusos/incusos-proxmox my-test.yaml # VM created in pool
|
||||
./incusos/incusos-proxmox --status my-test.yaml # Status scoped to pool
|
||||
```
|
||||
|
||||
### Step 11: (Optional) Full Lab Deployment
|
||||
|
||||
Once single-VM tests pass, deploy the full 4-VM lab:
|
||||
|
||||
|
|
@ -288,6 +350,144 @@ This deploys 1 Operations Center + 3 Incus nodes. After deployment:
|
|||
|
||||
---
|
||||
|
||||
## Lab Validation with lab-test
|
||||
|
||||
The `lab-test` script automates the end-to-end validation of deployed IncusOS
|
||||
VMs: single-node workloads, cluster formation, migration, and evacuation.
|
||||
|
||||
### Quick Start: Cluster Testing
|
||||
|
||||
```bash
|
||||
# 1. Check environment
|
||||
./incusos/incusos-proxmox --doctor
|
||||
|
||||
# 2. Deploy 3 nodes
|
||||
./incusos/incusos-proxmox --yes examples/lab-cluster.yaml
|
||||
|
||||
# 3. Add incus remotes for each node (use IPs from --status)
|
||||
./incusos/incusos-proxmox --status examples/lab-cluster.yaml
|
||||
incus remote add incus-lab-01 <IP1> --accept-certificate
|
||||
incus remote add incus-lab-02 <IP2> --accept-certificate
|
||||
incus remote add incus-lab-03 <IP3> --accept-certificate
|
||||
|
||||
# 4. Run full lab validation
|
||||
./incusos/lab-test --yes examples/lab-cluster.yaml
|
||||
```
|
||||
|
||||
### Phase-by-Phase Testing
|
||||
|
||||
```bash
|
||||
# Single-node workloads only (container + VM launch/exec)
|
||||
./incusos/lab-test --phase single --skip-deploy examples/lab-cluster.yaml
|
||||
|
||||
# Cluster formation only
|
||||
./incusos/lab-test --phase cluster --skip-deploy examples/lab-cluster.yaml
|
||||
|
||||
# Cluster workloads (scheduler placement + targeted placement)
|
||||
./incusos/lab-test --phase workload --skip-deploy examples/lab-cluster.yaml
|
||||
|
||||
# Migration and evacuation tests
|
||||
./incusos/lab-test --phase migrate --skip-deploy examples/lab-cluster.yaml
|
||||
```
|
||||
|
||||
### Manual Cluster Formation (Alternative)
|
||||
|
||||
If you prefer to form the cluster manually instead of using `lab-test`:
|
||||
|
||||
```bash
|
||||
# Step 1: Enable clustering on first node
|
||||
incus cluster enable incus-lab-01: incus-lab-01
|
||||
|
||||
# Step 2: Generate join token for second node
|
||||
incus cluster add incus-lab-01: incus-lab-02
|
||||
|
||||
# Step 3: Join second node
|
||||
incus cluster join incus-lab-01: incus-lab-02:
|
||||
|
||||
# Step 4: Repeat for third node
|
||||
incus cluster add incus-lab-01: incus-lab-03
|
||||
incus cluster join incus-lab-01: incus-lab-03:
|
||||
|
||||
# Step 5: Verify
|
||||
incus cluster list incus-lab-01:
|
||||
```
|
||||
|
||||
**Key points:**
|
||||
- `apply_defaults: true` on all nodes ensures matching storage pool and
|
||||
network bridge configurations (required for migration)
|
||||
- After joining, manage the cluster through the init node's remote
|
||||
- VM migration requires matching storage pool names across nodes
|
||||
|
||||
### Cleanup
|
||||
|
||||
```bash
|
||||
# Remove test instances only (keeps VMs and cluster)
|
||||
./incusos/lab-test --cleanup examples/lab-cluster.yaml
|
||||
|
||||
# Destroy VMs only
|
||||
./incusos/incusos-proxmox --cleanup examples/lab-cluster.yaml
|
||||
|
||||
# Full cleanup: VMs + ISOs + seeds + remotes + cache
|
||||
./incusos/incusos-proxmox --cleanup --deep --yes examples/lab-cluster.yaml
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Operations Center Lab
|
||||
|
||||
### Deploy
|
||||
|
||||
```bash
|
||||
./incusos/incusos-proxmox --yes examples/lab-oc.yaml
|
||||
./incusos/incusos-proxmox --status examples/lab-oc.yaml
|
||||
```
|
||||
|
||||
### Access the Web UI
|
||||
|
||||
```bash
|
||||
# Export client certificate for browser
|
||||
openssl pkcs12 -export \
|
||||
-inkey ~/.config/incus/client.key \
|
||||
-in ~/.config/incus/client.crt \
|
||||
-out client.pfx
|
||||
|
||||
# Import client.pfx into Firefox, navigate to https://<OC_IP>:8443
|
||||
```
|
||||
|
||||
### OC CLI Setup
|
||||
|
||||
```bash
|
||||
# Copy incus client certs for OC CLI
|
||||
mkdir -p ~/.config/operations-center
|
||||
cp ~/.config/incus/client.crt ~/.config/operations-center/
|
||||
cp ~/.config/incus/client.key ~/.config/operations-center/
|
||||
|
||||
# Add OC remote
|
||||
operations-center remote add lab-oc https://<OC_IP>:8443
|
||||
operations-center remote switch lab-oc
|
||||
|
||||
# Check for provisioning updates
|
||||
operations-center provisioning update list
|
||||
```
|
||||
|
||||
### Provisioning Workflow
|
||||
|
||||
```bash
|
||||
# Create a provisioning token
|
||||
operations-center provisioning token add --description "lab nodes" --uses 10
|
||||
|
||||
# Get provisioned ISO
|
||||
operations-center provisioning token get-image <token> ~/Downloads/IncusOS-provisioned.iso
|
||||
|
||||
# Redeploy nodes with provisioned ISO, or check if manual adoption works
|
||||
```
|
||||
|
||||
**Note:** Operations Center is under active development (v0.2.2+). Commands and
|
||||
APIs may change between versions. The `--doctor` command reports whether the OC
|
||||
CLI is installed.
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Likely Cause | Fix |
|
||||
|
|
@ -304,3 +504,6 @@ This deploys 1 Operations Center + 3 Incus nodes. After deployment:
|
|||
| "field server not found" YAML error | Wrong preseed structure | Use `preseed.certificates[]` not `preseed.server.certificates[]` |
|
||||
| Cert not auto-detected | No Incus keypair | Run `incus remote list` to generate one |
|
||||
| `genisoimage` not found | genisoimage not installed | Install `genisoimage` package |
|
||||
| Pool not found | Resource pool not created | Run `pveum pool add <name>` on Proxmox |
|
||||
| Re-run destroys VMs | Used option 3 in reconcile | Use option 1 (checks) or option 2 (continue) instead |
|
||||
| `--status` shows "not found" | VM was cleaned up | Deploy first, then check status |
|
||||
|
|
|
|||
|
|
@ -0,0 +1,44 @@
|
|||
# lab-cluster.yaml - 3-node IncusOS cluster for lab testing
|
||||
#
|
||||
# Deploys 3 standalone Incus nodes that can be clustered after deployment.
|
||||
# All nodes get apply_defaults: true so they have matching storage pool
|
||||
# and network bridge configurations.
|
||||
#
|
||||
# Usage:
|
||||
# incusos-proxmox --dry-run examples/lab-cluster.yaml # Preview
|
||||
# incusos-proxmox --yes examples/lab-cluster.yaml # Deploy
|
||||
# incusos-proxmox --status examples/lab-cluster.yaml # Check status
|
||||
# incusos-proxmox --cleanup examples/lab-cluster.yaml # Tear down VMs
|
||||
# incusos-proxmox --cleanup --deep examples/lab-cluster.yaml # Full cleanup
|
||||
#
|
||||
# After deployment, form a cluster with lab-test:
|
||||
# lab-test --phase cluster examples/lab-cluster.yaml
|
||||
|
||||
proxmox:
|
||||
host: 192.168.1.29
|
||||
method: api
|
||||
api_token_id: automation@pve!deploy
|
||||
storage: local-zfs
|
||||
pool: IncusLab
|
||||
|
||||
defaults:
|
||||
cores: 2
|
||||
memory: 4096
|
||||
disk: 50
|
||||
start_vmid: 900
|
||||
|
||||
vms:
|
||||
- name: incus-lab-01
|
||||
app: incus
|
||||
apply_defaults: true
|
||||
cores: 4
|
||||
memory: 8192
|
||||
disk: 64
|
||||
|
||||
- name: incus-lab-02
|
||||
app: incus
|
||||
apply_defaults: true
|
||||
|
||||
- name: incus-lab-03
|
||||
app: incus
|
||||
apply_defaults: true
|
||||
|
|
@ -0,0 +1,50 @@
|
|||
# lab-oc.yaml - Operations Center + 3-node lab
|
||||
#
|
||||
# Deploys 1 Operations Center instance and 3 Incus nodes.
|
||||
# Uses VMID range 910+ to avoid collisions with lab-cluster.yaml (900+).
|
||||
#
|
||||
# Workflow:
|
||||
# 1. Deploy with incusos-proxmox
|
||||
# 2. Access OC web UI (import client.pfx into browser)
|
||||
# 3. Create provisioning token in OC
|
||||
# 4. Either redeploy nodes with OC-provisioned ISO or manually adopt
|
||||
# 5. Form cluster via OC CLI or web UI
|
||||
#
|
||||
# Usage:
|
||||
# incusos-proxmox --dry-run examples/lab-oc.yaml
|
||||
# incusos-proxmox --yes examples/lab-oc.yaml
|
||||
# incusos-proxmox --status examples/lab-oc.yaml
|
||||
# incusos-proxmox --cleanup --deep examples/lab-oc.yaml
|
||||
|
||||
proxmox:
|
||||
host: 192.168.1.29
|
||||
method: api
|
||||
api_token_id: automation@pve!deploy
|
||||
storage: local-zfs
|
||||
pool: IncusLab
|
||||
|
||||
defaults:
|
||||
cores: 2
|
||||
memory: 4096
|
||||
disk: 50
|
||||
start_vmid: 910
|
||||
|
||||
vms:
|
||||
- name: lab-oc
|
||||
app: operations-center
|
||||
apply_defaults: true
|
||||
|
||||
- name: lab-node-01
|
||||
app: incus
|
||||
apply_defaults: true
|
||||
cores: 4
|
||||
memory: 8192
|
||||
disk: 64
|
||||
|
||||
- name: lab-node-02
|
||||
app: incus
|
||||
apply_defaults: true
|
||||
|
||||
- name: lab-node-03
|
||||
app: incus
|
||||
apply_defaults: true
|
||||
|
|
@ -3,15 +3,27 @@ proxmox:
|
|||
method: api
|
||||
api_token_id: automation@pve!deploy
|
||||
storage: local-zfs # VM disk storage (default: local-lvm)
|
||||
pool: IncusLab
|
||||
|
||||
defaults:
|
||||
cores: 2
|
||||
memory: 4096 # Minimum for IncusOS
|
||||
disk: 50 # Minimum for IncusOS
|
||||
start_vmid: 900 # High range to avoid collisions
|
||||
|
||||
vms:
|
||||
- name: incus-test
|
||||
- name: incus-lab-01
|
||||
app: incus
|
||||
apply_defaults: true
|
||||
cores: 4
|
||||
memory: 8192
|
||||
disk: 64
|
||||
|
||||
- name: incus-lab-02
|
||||
app: incus
|
||||
apply_defaults: true
|
||||
|
||||
- name: ops-center
|
||||
app: operations-center
|
||||
apply_defaults: true
|
||||
|
||||
|
|
@ -15,6 +15,7 @@ proxmox:
|
|||
storage: local-lvm # VM disk storage
|
||||
iso_storage: local # ISO/image storage
|
||||
bridge: vmbr0 # Network bridge
|
||||
# pool: IncusLab # Resource pool (optional, for isolation)
|
||||
method: ssh # ssh (default) or api
|
||||
ssh_user: root # SSH user for ssh method
|
||||
# api_token_id: automation@pve!deploy # For api method
|
||||
|
|
|
|||
|
|
@ -8,6 +8,7 @@
|
|||
|
||||
proxmox:
|
||||
host: 192.168.1.10 # Change to your Proxmox host
|
||||
# pool: IncusLab # Resource pool (optional, for isolation)
|
||||
# All other settings use defaults:
|
||||
# node: pve, storage: local-lvm, iso_storage: local,
|
||||
# bridge: vmbr0, method: ssh, ssh_user: root
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
Loading…
Reference in New Issue