Add advanced migration testing, cluster lifecycle, and clustering guide
- New lab-advanced.yaml (3-node 6/4/4 core heterogeneous cluster) and lab-replace.yaml (single replacement node) for advanced migration and cluster lifecycle testing - New notes/clustering-guide.md with full clustering walkthrough: cluster formation, storage pool conflict fix, join workflow, VM live migration with limits.cpu range fix, multi-vCPU test results (2/3/4 vCPUs), edge cases (I/O, memory, concurrent), and node replacement lifecycle - Updated CLAUDE.md with clustering docs, rebalancing config keys, cluster enable syntax fix (TWO args not one), cluster remove syntax, agent reconnect timing (3-4s not 1-2s), and lifecycle procedure - proxmox.yaml.example for Proxmox connection config reference Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
2aca6fb33b
commit
732d70b38e
|
|
@ -25,3 +25,6 @@ Thumbs.db
|
|||
|
||||
# Go binaries (flasher-tool)
|
||||
flasher-tool
|
||||
|
||||
# Proxmox connection config (contains credentials)
|
||||
proxmox.yaml
|
||||
|
|
|
|||
226
CLAUDE.md
226
CLAUDE.md
|
|
@ -19,9 +19,11 @@ incu-contrib/
|
|||
│ ├── incusos-seed # Seed archive generator
|
||||
│ ├── incusos-proxmox # Declarative Proxmox VM deployment
|
||||
│ ├── lab-test # Guided lab validation (workloads, clustering, migration)
|
||||
│ ├── proxmox.yaml # Proxmox connection config (gitignored, contains credentials)
|
||||
│ ├── TESTING.md # Testing guide for incusos-proxmox and lab-test
|
||||
│ └── examples/ # Example seed + Proxmox YAML files
|
||||
└── notes/ # Research notes and reference material
|
||||
└── clustering-guide.md # Detailed Incus clustering walkthrough
|
||||
```
|
||||
|
||||
## Key technical context
|
||||
|
|
@ -78,6 +80,11 @@ incu-contrib/
|
|||
- **`incusos-proxmox`** reads a YAML config, generates per-VM SEED_DATA images
|
||||
via `incusos-seed --format iso`, uploads the ISO + seeds to Proxmox, creates
|
||||
VMs with IncusOS-correct settings, and boots them through installation.
|
||||
- **Config separation**: Proxmox connection settings (host, credentials, pool)
|
||||
live in `incusos/proxmox.yaml` (gitignored). Lab configs (`lab-cluster.yaml`,
|
||||
etc.) only define VM specs. Merge priority: `proxmox.yaml` base → lab config
|
||||
`proxmox:` overlay → CLI flags (`--host`, `--method`). Auto-discovery looks
|
||||
for `proxmox.yaml` in script directory then cwd; override with `--proxmox FILE`.
|
||||
- **Connection methods**: SSH (default, `ssh root@host qm ...`) or API
|
||||
(`curl -k https://host:8006/api2/json/...` with `PVEAPIToken` header).
|
||||
- **Minimum API privileges** for token-based access:
|
||||
|
|
@ -142,27 +149,198 @@ incu-contrib/
|
|||
|
||||
- **Cluster formation** is done entirely through the `incus` CLI using remotes.
|
||||
No SSH to the IncusOS nodes is needed (IncusOS is immutable, no shell access).
|
||||
- **Workflow:**
|
||||
- **No VIP needed**: each node advertises its own IP as its cluster address.
|
||||
Clients can connect to any cluster member; requests are forwarded internally.
|
||||
|
||||
#### Pre-clustering: fix core.https_address
|
||||
|
||||
- IncusOS nodes default to `core.https_address: :8443` (wildcard / all
|
||||
interfaces). Clustering requires a **specific routable IP** so nodes can
|
||||
address each other.
|
||||
- **Set the IP on every node BEFORE enabling clustering:**
|
||||
```bash
|
||||
# Enable clustering on first node
|
||||
incus cluster enable <remote1>: <member-name>
|
||||
|
||||
# Generate join token (on cluster, for new member name)
|
||||
incus cluster add <remote1>: <new-member-name>
|
||||
|
||||
# Join second node to cluster
|
||||
incus cluster join <remote1>: <remote2>:
|
||||
|
||||
# Verify
|
||||
incus cluster list <remote1>:
|
||||
incus config set <remote>: core.https_address <NODE_IP>:8443
|
||||
```
|
||||
- **`apply_defaults: true`** must be set on all nodes that will be clustered --
|
||||
this ensures matching storage pool (`local`) and network bridge names, which
|
||||
are required for migration.
|
||||
- Get each node's routable IP via the API:
|
||||
```bash
|
||||
incus query <remote>:/1.0 | python3 -c "import sys,json; d=json.load(sys.stdin); \
|
||||
[print(a) for a in d['environment']['addresses'] \
|
||||
if not a.startswith('10.') and not a.startswith('fd42:') and not a.startswith('[')]"
|
||||
```
|
||||
- This is safe to do while remotes are connected -- the remote already points
|
||||
to the specific IP; we're just narrowing the bind address. Certificate trust
|
||||
is fingerprint-based, not address-based.
|
||||
|
||||
#### Cluster enable (init node)
|
||||
|
||||
- ```bash
|
||||
incus cluster enable <remote>: <member-name>
|
||||
```
|
||||
Note: this is TWO arguments: `<remote>:` (trailing colon) and `<member-name>`.
|
||||
The help text shows `[<remote>:] <name>` — NOT `remote:name` as a single arg.
|
||||
- **TLS certificate regeneration**: enabling clustering causes the server to
|
||||
generate a new TLS certificate (cluster cert). The new cert may only have
|
||||
SANs for `127.0.0.1` and `::1`, breaking the existing remote.
|
||||
- **Fix**: remove and re-add the remote to pin the new certificate:
|
||||
```bash
|
||||
incus remote switch local # if init remote is current default
|
||||
incus remote remove <remote>
|
||||
incus remote add <remote> https://<NODE_IP>:8443 --accept-certificate
|
||||
```
|
||||
- The cert trust on the server side (client → server) is unaffected -- it's
|
||||
stored by fingerprint in the Incus database, independent of listen address.
|
||||
|
||||
#### Joining nodes: the apply_defaults storage pool conflict
|
||||
|
||||
- **`apply_defaults: true`** creates a `local` ZFS storage pool and `incusbr0`
|
||||
network bridge on each node at install time. When joining a cluster that
|
||||
already has a `local` pool defined, the join wizard asks for `source` and
|
||||
`zfs.pool_name` properties but then tries to set them as cluster-wide config.
|
||||
Incus rejects this because they are **member-specific** keys.
|
||||
- **The fix**: delete the storage pool and network on the joining node before
|
||||
joining. The join process then creates the member-specific pool entry
|
||||
correctly.
|
||||
```bash
|
||||
# 1. Remove config references
|
||||
incus config unset <remote>: storage.backups_volume
|
||||
incus config unset <remote>: storage.images_volume
|
||||
# 2. Delete volumes
|
||||
incus storage volume delete <remote>:local backups
|
||||
incus storage volume delete <remote>:local images
|
||||
# 3. Clear default profile references (pool is "in use" otherwise)
|
||||
incus profile device remove <remote>:default root
|
||||
incus profile device remove <remote>:default eth0
|
||||
# 4. Delete pool and network
|
||||
incus storage delete <remote>:local
|
||||
incus network delete <remote>:incusbr0
|
||||
```
|
||||
|
||||
#### Join workflow
|
||||
|
||||
- **Generate token** (on init node, single argument `remote:member-name`):
|
||||
```bash
|
||||
incus cluster add <init-remote>:<new-member-name>
|
||||
```
|
||||
- **Join** (interactive -- prompts for 5 values):
|
||||
```bash
|
||||
incus cluster join <init-remote>: <joining-remote>:
|
||||
```
|
||||
Interactive prompts and correct answers:
|
||||
1. IP address → accept default (node's IP, already set via core.https_address)
|
||||
2. Member name → accept default (matches the token)
|
||||
3. "All existing data is lost" → `yes`
|
||||
4. `source` property for storage pool "local" → `local/incus`
|
||||
5. `zfs.pool_name` property for storage pool "local" → `local/incus`
|
||||
- **Automated (non-interactive):**
|
||||
```bash
|
||||
printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join <init-remote>: <joining-remote>:
|
||||
```
|
||||
- **After join**: the joining node gets a new cluster certificate. Fix the
|
||||
remote (same as init node):
|
||||
```bash
|
||||
incus remote remove <joining-remote>
|
||||
incus remote add <joining-remote> https://<NODE_IP>:8443 --accept-certificate
|
||||
```
|
||||
|
||||
#### Command syntax gotchas
|
||||
|
||||
- `incus cluster enable remote: member-name` -- TWO arguments (remote: + name)
|
||||
- `incus cluster add remote:member-name` -- ONE argument (no space)
|
||||
- `incus cluster remove remote:member-name --force` -- ONE argument; prompts
|
||||
"yes/no" even with `--force`, pipe `printf "yes\n"` for automation
|
||||
- `incus cluster evacuate remote:member-name` -- ONE argument (no space)
|
||||
- `incus cluster restore remote:member-name` -- ONE argument (no space)
|
||||
- `incus cluster join init-remote: joining-remote:` -- TWO arguments (space)
|
||||
- `incus storage show remote:pool` -- ONE argument (no space)
|
||||
- `incus storage show remote:pool --target member` -- target flag for
|
||||
member-specific config
|
||||
- `incus config set remote: key value` -- remote with trailing colon + space
|
||||
- General rule: `remote:resource` for targeting a resource, `remote:` (trailing
|
||||
colon) for targeting the server itself
|
||||
|
||||
#### Post-join state
|
||||
|
||||
- After joining, the cluster is managed through the init node's remote. The
|
||||
individual node remotes still work for node-specific operations.
|
||||
- **`lab-test`** automates cluster formation, workload testing, and migration.
|
||||
|
||||
#### Workload placement and migration
|
||||
|
||||
- **Targeted launch**: `incus launch images:debian/12 <cluster-remote>:name --target <member>`
|
||||
- **Cluster-wide visibility**: `incus list` on any member shows all instances.
|
||||
- **Container migration**: stop/move/start only (CRIU live migration is
|
||||
unreliable). Data persists, processes do not.
|
||||
```bash
|
||||
incus stop <remote>:<instance>
|
||||
incus move <remote>:<instance> --target <destination-member>
|
||||
incus start <remote>:<instance>
|
||||
```
|
||||
- **VM live migration**: requires `migration.stateful=true` (must be set
|
||||
while VM is stopped). Preserves running state with no downtime.
|
||||
```bash
|
||||
incus move <remote>:<instance> --target <destination-member>
|
||||
```
|
||||
- **VM live migration requires `limits.cpu` as a range** (e.g., `0-1`),
|
||||
not an integer. Without this, Incus sets QEMU's `maxcpus` to the host's
|
||||
CPU count (`driver_qemu_templates.go`: `maxcpus = min(cpu.Total, 64)`).
|
||||
Different `maxcpus` values size the ICH9 ACPI CPU hotplug state arrays
|
||||
differently, causing `Missing section footer for ICH9LPC` on restore.
|
||||
Using a range (pinning syntax) eliminates `maxcpus` entirely and uses
|
||||
fixed `sockets/cores/threads` topology — portable across all hosts.
|
||||
```bash
|
||||
# WRONG: integer → maxcpus varies by host → migration fails
|
||||
incus config set <instance> limits.cpu=2
|
||||
# RIGHT: range → fixed topology → migration works everywhere
|
||||
incus config set <instance> limits.cpu=0-1
|
||||
```
|
||||
- **VM live migration works in nested virtualization** (IncusOS inside
|
||||
Proxmox on Intel). It is NOT limited to bare metal. Tested with QEMU
|
||||
10.2.1 on Intel i9-13900HK with heterogeneous host core counts (4 vs 2).
|
||||
- **The `vnmi` CPUID warning** (`CPUID[eax=8000000Ah].EDX.vnmi`) that
|
||||
appears during migration is cosmetic. It fires from QEMU's feature
|
||||
dependency checker before KVM filters out unsupported features and does
|
||||
not affect migration.
|
||||
- **Stateful stop/restore** (`incus stop --stateful` + `incus start`) also
|
||||
requires the `limits.cpu` range fix. Use `incus start --stateless` to
|
||||
discard a saved state file that cannot be restored.
|
||||
- **VM `size.state` config**: stateful operations require `size.state` on
|
||||
the root disk (`incus config device add <instance> root disk path=/
|
||||
pool=local size.state=2GiB`). Without it, `incus stop --stateful` fails.
|
||||
- **Cluster evacuation**: `incus cluster evacuate <remote>:<member> --force`
|
||||
(ONE argument, like `cluster enable` and `cluster add`).
|
||||
Use `--action stop` if VMs lack the `limits.cpu` range fix.
|
||||
Restore with `incus cluster restore <remote>:<member> --force`.
|
||||
- **VM agent reconnect**: after live migration, the incus agent inside the VM
|
||||
needs ~3-4 seconds to reconnect. `incus exec` commands issued immediately
|
||||
after migration may fail with "VM agent isn't currently running". Scripts
|
||||
should `sleep 4` after migration before running `incus exec`.
|
||||
- **Multi-vCPU migration**: tested with 2, 3, and 4 vCPU VMs across
|
||||
heterogeneous hosts (6/4/4 cores). Odd vCPU counts (e.g., `limits.cpu=0-2`)
|
||||
work identically to even counts. A 4-vCPU VM on a 4-core host (100% core
|
||||
usage) migrates without issues. `size.state=4GiB` recommended for 3-4 vCPU VMs.
|
||||
- **Concurrent migrations**: migrating multiple VMs simultaneously from
|
||||
different source nodes works without interference. ~140 MB/s per migration.
|
||||
- **Active I/O during migration**: disk writes and network activity survive
|
||||
live migration transparently. File integrity verified after migration.
|
||||
- **Cluster rebalancing**: Incus can auto-redistribute VMs when a new node
|
||||
joins. Only moves VMs with `migration.stateful=true`. Containers are NOT
|
||||
auto-rebalanced.
|
||||
```bash
|
||||
incus config set <remote>: cluster.rebalance.interval=1 # minutes
|
||||
incus config set <remote>: cluster.rebalance.threshold=10 # imbalance %
|
||||
incus config set <remote>: cluster.rebalance.batch=2 # max VMs/run
|
||||
incus config set <remote>: cluster.rebalance.cooldown=5m # wait between runs
|
||||
```
|
||||
- **Node replacement lifecycle**: evacuate → remove → destroy → deploy fresh →
|
||||
join → auto-rebalance. Full procedure tested. See `notes/clustering-guide.md`
|
||||
for step-by-step instructions.
|
||||
- **`incus cluster remove` requires confirmation**: even with `--force`, it
|
||||
prompts "Are you really sure?". Pipe `yes` for automation:
|
||||
```bash
|
||||
printf "yes\n" | incus cluster remove <remote>:<member> --force
|
||||
```
|
||||
- See `notes/clustering-guide.md` for full details and references.
|
||||
|
||||
### Lab validation (lab-test)
|
||||
|
||||
- **`lab-test`** reads the same YAML config as `incusos-proxmox` and operates
|
||||
|
|
@ -187,11 +365,19 @@ incu-contrib/
|
|||
### incusos-proxmox doctor and cleanup
|
||||
|
||||
- **`--doctor`**: standalone environment check. No config file required.
|
||||
Checks tool versions, IncusOS CDN, and optionally Proxmox connectivity.
|
||||
- **`--cleanup --deep`**: destroys VMs + deletes ISOs/seeds from Proxmox +
|
||||
removes incus remotes + clears local ISO cache.
|
||||
- **`--cleanup-all`**: pool-wide cleanup. Destroys all VMs with the
|
||||
`[incusos-lab:managed]` marker in the configured resource pool.
|
||||
Checks tool versions, IncusOS CDN, proxmox.yaml discovery, and optionally
|
||||
Proxmox connectivity (from proxmox.yaml or config file).
|
||||
- **`--cleanup`**: destroys VMs defined in the config file.
|
||||
- **`--cleanup --deep`**: also deletes the specific IncusOS ISO used by this
|
||||
deployment + per-VM seed ISOs + incus remotes + local cache. Does NOT delete
|
||||
all IncusOS ISOs (unlike the old behavior).
|
||||
- **`--cleanup-all`**: pool-wide cleanup. Only needs `proxmox.yaml` (no lab
|
||||
config required). Destroys all VMs with `[incusos-lab:managed]` marker.
|
||||
- **`--cleanup-all --deep`**: aggressive blanket delete of ALL `IncusOS_*.iso`
|
||||
and `seed-*.iso` from storage + remotes + cache.
|
||||
- **`--verbose` / `-v`**: shows detailed output (tool paths, API calls). Default
|
||||
output is concise (step names + results). `--quiet` suppresses everything
|
||||
except warnings and errors.
|
||||
|
||||
## Coding conventions for scripts
|
||||
|
||||
|
|
|
|||
|
|
@ -296,13 +296,16 @@ This script ensures all settings are correct, preventing install failures.
|
|||
--no-cert Skip certificate injection
|
||||
--vm NAME Operate on a single VM only
|
||||
-y, --yes Skip confirmation prompts
|
||||
--proxmox FILE Proxmox connection config (default: auto-discover)
|
||||
--doctor Check tool versions and environment health
|
||||
--status Check deployment status (shortcut for --phase status)
|
||||
--cleanup Destroy VMs defined in config
|
||||
--cleanup --deep Also remove ISOs, seeds, remotes, and local cache
|
||||
--cleanup-all Destroy ALL managed VMs in the resource pool
|
||||
--cleanup --deep Also remove deployment ISO, seeds, remotes, and cache
|
||||
--cleanup-all Destroy ALL managed VMs in pool (no config file needed)
|
||||
--cleanup-all --deep Also remove ALL ISOs and seeds from storage
|
||||
--force-cleanup Destroy VMs even without management marker
|
||||
--dry-run Preview without executing
|
||||
-v, --verbose Show detailed output (tool paths, API calls)
|
||||
-q, --quiet Suppress info output
|
||||
-h, --help Full help text
|
||||
-V, --version Show version
|
||||
|
|
@ -314,14 +317,15 @@ Run `--doctor` to verify your environment without a config file:
|
|||
|
||||
```bash
|
||||
./incusos-proxmox --doctor
|
||||
./incusos-proxmox --doctor -v # verbose: show tool paths
|
||||
```
|
||||
|
||||
This checks: incus client, flasher-tool, operations-center CLI (optional),
|
||||
python3, curl, genisoimage, jq, incusos-seed sibling, and IncusOS CDN
|
||||
availability.
|
||||
python3, curl, genisoimage, jq, incusos-seed sibling, IncusOS CDN
|
||||
availability, and `proxmox.yaml` discovery.
|
||||
|
||||
If a config file is also provided, additionally checks Proxmox connectivity,
|
||||
pool existence, and API token validity:
|
||||
If `proxmox.yaml` is found or a config file is provided, additionally checks
|
||||
Proxmox connectivity, pool existence, and API token validity:
|
||||
|
||||
```bash
|
||||
./incusos-proxmox --doctor examples/lab-cluster.yaml
|
||||
|
|
@ -329,22 +333,28 @@ pool existence, and API token validity:
|
|||
|
||||
### Deep Cleanup
|
||||
|
||||
Use `--deep` with `--cleanup` or `--cleanup-all` to remove all artifacts:
|
||||
Use `--deep` with `--cleanup` or `--cleanup-all` to remove artifacts beyond VMs:
|
||||
|
||||
```bash
|
||||
# VMs + ISOs + seeds + incus remotes + local cache
|
||||
# VMs + the IncusOS ISO used by this deployment + seeds + remotes + cache
|
||||
./incusos-proxmox --cleanup --deep examples/lab-cluster.yaml
|
||||
|
||||
# Pool-wide: destroy all managed VMs + artifacts
|
||||
./incusos-proxmox --cleanup-all --deep examples/lab-cluster.yaml
|
||||
# Pool-wide: destroy all managed VMs (uses proxmox.yaml, no config needed)
|
||||
./incusos-proxmox --cleanup-all
|
||||
./incusos-proxmox --cleanup-all --deep # also nuke ALL ISOs + seeds
|
||||
```
|
||||
|
||||
`--deep` additionally:
|
||||
- Deletes seed ISOs from Proxmox storage
|
||||
- Deletes IncusOS ISOs from Proxmox storage
|
||||
`--cleanup --deep`:
|
||||
- Deletes per-VM seed ISOs from Proxmox storage
|
||||
- Deletes the specific IncusOS ISO used by this deployment (not all of them)
|
||||
- Removes matching `incus remote` entries
|
||||
- Clears the local ISO cache (`~/.cache/incusos-proxmox/iso-cache/`)
|
||||
|
||||
`--cleanup-all --deep`:
|
||||
- Deletes ALL `IncusOS_*.iso` and `seed-*.iso` from storage (intentionally aggressive)
|
||||
- Removes remotes for the destroyed VMs
|
||||
- Clears the local ISO cache
|
||||
|
||||
### Deployment Phases
|
||||
|
||||
| Phase | What it does |
|
||||
|
|
@ -356,18 +366,41 @@ Use `--deep` with `--cleanup` or `--cleanup-all` to remove all artifacts:
|
|||
| `install` | Start VMs, detect install completion via disk I/O monitoring, detach CD-ROMs, boot from disk |
|
||||
| `status` | Check deployment status: Proxmox state, IP, port 8443, incus remotes, post-deploy checks |
|
||||
|
||||
### Config File Format
|
||||
### Proxmox Connection Config
|
||||
|
||||
Proxmox connection settings live in a separate `proxmox.yaml` file to avoid
|
||||
duplicating credentials across lab configs. Auto-discovered in the script
|
||||
directory and current directory. Override with `--proxmox FILE`.
|
||||
|
||||
Copy the template to get started:
|
||||
|
||||
```bash
|
||||
cp incusos/examples/proxmox.yaml.example incusos/proxmox.yaml
|
||||
# Edit with your Proxmox host, credentials, and storage settings
|
||||
```
|
||||
|
||||
```yaml
|
||||
# proxmox.yaml (flat structure, no wrapping key)
|
||||
host: 192.168.1.10
|
||||
method: api
|
||||
api_token_id: automation@pve!deploy
|
||||
node: pve
|
||||
storage: local-lvm
|
||||
iso_storage: local
|
||||
bridge: vmbr0
|
||||
pool: IncusLab
|
||||
ssh_user: root
|
||||
```
|
||||
|
||||
Lab configs can override any setting by including a `proxmox:` section.
|
||||
Merge priority: `proxmox.yaml` → lab config `proxmox:` → CLI flags.
|
||||
|
||||
### Lab Config File Format
|
||||
|
||||
```yaml
|
||||
# proxmox: section is optional — overrides proxmox.yaml if present
|
||||
proxmox:
|
||||
host: 192.168.1.10 # Proxmox host (required)
|
||||
node: pve # Node name (default: pve)
|
||||
storage: local-lvm # VM disk storage (default: local-lvm)
|
||||
iso_storage: local # ISO storage (default: local)
|
||||
bridge: vmbr0 # Network bridge (default: vmbr0)
|
||||
pool: IncusLab # Resource pool (optional, for isolation)
|
||||
method: ssh # ssh (default) or api
|
||||
ssh_user: root # For ssh method
|
||||
storage: local-zfs # Override storage for this deployment
|
||||
|
||||
defaults:
|
||||
cores: 4
|
||||
|
|
@ -602,10 +635,11 @@ See the [`examples/`](examples/) directory for sample seed configurations:
|
|||
- [`incus-cluster-node.yaml`](examples/incus-cluster-node.yaml) -- Cluster node (no defaults)
|
||||
- [`ops-center.yaml`](examples/ops-center.yaml) -- Operations Center
|
||||
- [`network-static.yaml`](examples/network-static.yaml) -- Static network configuration
|
||||
- [`proxmox-lab.yaml`](examples/proxmox-lab.yaml) -- Proxmox: 4-VM lab (ops-center + 3-node cluster)
|
||||
- [`proxmox-minimal.yaml`](examples/proxmox-minimal.yaml) -- Proxmox: single standalone VM
|
||||
- [`lab-cluster.yaml`](examples/lab-cluster.yaml) -- Proxmox: 3-node cluster for lab validation
|
||||
- [`lab-oc.yaml`](examples/lab-oc.yaml) -- Proxmox: Operations Center + 3 nodes
|
||||
- [`proxmox.yaml.example`](examples/proxmox.yaml.example) -- Proxmox connection config template
|
||||
- [`proxmox-lab.yaml`](examples/proxmox-lab.yaml) -- Proxmox: 4-VM lab (self-contained, includes proxmox: section)
|
||||
- [`proxmox-minimal.yaml`](examples/proxmox-minimal.yaml) -- Proxmox: single standalone VM (self-contained)
|
||||
- [`lab-cluster.yaml`](examples/lab-cluster.yaml) -- Proxmox: 3-node cluster (uses proxmox.yaml)
|
||||
- [`lab-oc.yaml`](examples/lab-oc.yaml) -- Proxmox: Operations Center + 3 nodes (uses proxmox.yaml)
|
||||
|
||||
## Security Notes
|
||||
|
||||
|
|
|
|||
|
|
@ -129,13 +129,23 @@ ssh root@<YOUR_PROXMOX_IP> pvesh get /version
|
|||
You should see JSON with the Proxmox VE version. If this fails, fix SSH
|
||||
key auth first (`ssh-copy-id root@<host>`).
|
||||
|
||||
### Step 2: Create a Test Config
|
||||
### Step 2: Set Up Proxmox Connection
|
||||
|
||||
```bash
|
||||
cp incusos/examples/proxmox.yaml.example incusos/proxmox.yaml
|
||||
```
|
||||
|
||||
Edit `incusos/proxmox.yaml` with your Proxmox host settings. This is
|
||||
auto-discovered by the scripts and avoids repeating credentials in each
|
||||
lab config.
|
||||
|
||||
### Step 3: Create a Test Config
|
||||
|
||||
```bash
|
||||
cp incusos/examples/proxmox-minimal.yaml my-test.yaml
|
||||
```
|
||||
|
||||
Edit `my-test.yaml` -- change only the host IP:
|
||||
Edit `my-test.yaml` -- change only the host IP (or rely on proxmox.yaml):
|
||||
|
||||
```yaml
|
||||
proxmox:
|
||||
|
|
@ -155,7 +165,7 @@ vms:
|
|||
|
||||
Using VMID 900+ avoids clashing with existing VMs.
|
||||
|
||||
### Step 3: Dry Run
|
||||
### Step 4: Dry Run
|
||||
|
||||
```bash
|
||||
./incusos/incusos-proxmox --dry-run my-test.yaml
|
||||
|
|
@ -172,7 +182,7 @@ Using VMID 900+ avoids clashing with existing VMs.
|
|||
- `[dry-run]` prefix on all actions -- nothing actually executed
|
||||
- No errors
|
||||
|
||||
### Step 4: Deploy (Single VM)
|
||||
### Step 5: Deploy (Single VM)
|
||||
|
||||
```bash
|
||||
./incusos/incusos-proxmox my-test.yaml
|
||||
|
|
@ -196,7 +206,7 @@ Using VMID 900+ avoids clashing with existing VMs.
|
|||
disk, and reboot
|
||||
- After reboot you should see the IncusOS login prompt
|
||||
|
||||
### Step 5: Check Deployment Status
|
||||
### Step 6: Check Deployment Status
|
||||
|
||||
After the deploy completes, verify with the built-in status command:
|
||||
|
||||
|
|
@ -212,7 +222,7 @@ After the deploy completes, verify with the built-in status command:
|
|||
- Port 8443 is open
|
||||
- Incus remote status shown (if configured)
|
||||
|
||||
### Step 6: Verify the Deployed VM
|
||||
### Step 7: Verify the Deployed VM
|
||||
|
||||
Once the script reports success with an IP address:
|
||||
|
||||
|
|
@ -233,7 +243,7 @@ incus network list incus-test:
|
|||
ARP-based lookup (MAC from Proxmox VM config → ARP table). If the IP is wrong,
|
||||
check the Proxmox console — stale ARP entries can be misleading.
|
||||
|
||||
### Step 7: Test Re-run Reconcile
|
||||
### Step 8: Test Re-run Reconcile
|
||||
|
||||
Re-run the deploy to verify the reconcile menu:
|
||||
|
||||
|
|
@ -252,7 +262,7 @@ Also test with `--yes` to verify it defaults to safe behavior:
|
|||
|
||||
**Verify:** Runs post-deployment checks, does NOT destroy.
|
||||
|
||||
### Step 8: Clean Up the Test VM
|
||||
### Step 9: Clean Up the Test VM
|
||||
|
||||
```bash
|
||||
./incusos/incusos-proxmox --cleanup my-test.yaml
|
||||
|
|
@ -269,7 +279,7 @@ Remove the remote too:
|
|||
incus remote remove incus-test
|
||||
```
|
||||
|
||||
### Step 9: (Optional) Test API Method
|
||||
### Step 10: (Optional) Test API Method
|
||||
|
||||
After single-VM success with SSH, switch to API tokens.
|
||||
|
||||
|
|
@ -302,9 +312,9 @@ proxmox:
|
|||
api_token_id: automation@pve!deploy
|
||||
```
|
||||
|
||||
Repeat steps 3-6 using the API method.
|
||||
Repeat steps 4-7 using the API method.
|
||||
|
||||
### Step 10: (Optional) Test with Resource Pool
|
||||
### Step 11: (Optional) Test with Resource Pool
|
||||
|
||||
For API method with pool isolation:
|
||||
|
||||
|
|
@ -331,7 +341,7 @@ Deploy and verify:
|
|||
./incusos/incusos-proxmox --status my-test.yaml # Status scoped to pool
|
||||
```
|
||||
|
||||
### Step 11: (Optional) Full Lab Deployment
|
||||
### Step 12: (Optional) Full Lab Deployment
|
||||
|
||||
Once single-VM tests pass, deploy the full 4-VM lab:
|
||||
|
||||
|
|
@ -392,32 +402,229 @@ incus remote add incus-lab-03 <IP3> --accept-certificate
|
|||
|
||||
### Manual Cluster Formation (Alternative)
|
||||
|
||||
If you prefer to form the cluster manually instead of using `lab-test`:
|
||||
If you prefer to form the cluster manually instead of using `lab-test`,
|
||||
follow the steps below. See also `notes/clustering-guide.md` for the full
|
||||
reference guide with explanations of what happens under the hood.
|
||||
|
||||
#### Prerequisites
|
||||
|
||||
```bash
|
||||
# Step 1: Enable clustering on first node
|
||||
incus cluster enable incus-lab-01: incus-lab-01
|
||||
# Check client version (need 6.20+ for remote cluster join)
|
||||
incus version
|
||||
|
||||
# Step 2: Generate join token for second node
|
||||
# Verify all three remotes are working
|
||||
incus info incus-lab-01: | head -5
|
||||
incus info incus-lab-02: | head -5
|
||||
incus info incus-lab-03: | head -5
|
||||
```
|
||||
|
||||
#### Step 1: Set specific IPs on all nodes
|
||||
|
||||
IncusOS defaults to `core.https_address: :8443` (wildcard). Clustering
|
||||
requires a specific routable IP on each node.
|
||||
|
||||
```bash
|
||||
# Discover each node's routable IP
|
||||
for node in incus-lab-01 incus-lab-02 incus-lab-03; do
|
||||
echo "=== $node ==="
|
||||
incus query "$node:/1.0" | python3 -c "import sys,json; d=json.load(sys.stdin); \
|
||||
[print(a) for a in d['environment']['addresses'] \
|
||||
if not a.startswith('10.') and not a.startswith('fd42:') and not a.startswith('[')]"
|
||||
done
|
||||
|
||||
# Set the specific IP on each node (replace with actual IPs)
|
||||
incus config set incus-lab-01: core.https_address <IP1>:8443
|
||||
incus config set incus-lab-02: core.https_address <IP2>:8443
|
||||
incus config set incus-lab-03: core.https_address <IP3>:8443
|
||||
|
||||
# Verify connectivity still works after each change
|
||||
incus info incus-lab-01: | head -3
|
||||
incus info incus-lab-02: | head -3
|
||||
incus info incus-lab-03: | head -3
|
||||
```
|
||||
|
||||
#### Step 2: Enable clustering on the init node
|
||||
|
||||
```bash
|
||||
incus cluster enable incus-lab-01:incus-lab-01
|
||||
```
|
||||
|
||||
> **Important**: This regenerates the server's TLS certificate. Your remote
|
||||
> will break with a certificate error. Fix it:
|
||||
|
||||
```bash
|
||||
incus remote switch local # if incus-lab-01 is the current default
|
||||
incus remote remove incus-lab-01
|
||||
incus remote add incus-lab-01 https://<IP1>:8443 --accept-certificate
|
||||
```
|
||||
|
||||
Verify:
|
||||
|
||||
```bash
|
||||
incus cluster list incus-lab-01:
|
||||
# Should show one member (incus-lab-01), status ONLINE, role database-leader
|
||||
```
|
||||
|
||||
#### Step 3: Prepare joining nodes
|
||||
|
||||
Nodes with `apply_defaults: true` have a pre-existing `local` storage pool
|
||||
and `incusbr0` network. These conflict with the cluster join process (the
|
||||
join wizard tries to set member-specific ZFS config keys at cluster level).
|
||||
Delete them before joining:
|
||||
|
||||
```bash
|
||||
# For each joining node (incus-lab-02, incus-lab-03):
|
||||
NODE=incus-lab-02 # then repeat with incus-lab-03
|
||||
|
||||
# Remove server-level references to the pool
|
||||
incus config unset "$NODE": storage.backups_volume
|
||||
incus config unset "$NODE": storage.images_volume
|
||||
|
||||
# Delete the backup and image volumes
|
||||
incus storage volume delete "$NODE":local backups
|
||||
incus storage volume delete "$NODE":local images
|
||||
|
||||
# Clear default profile references (otherwise pool is "in use")
|
||||
incus profile device remove "$NODE":default root
|
||||
incus profile device remove "$NODE":default eth0
|
||||
|
||||
# Delete pool and network
|
||||
incus storage delete "$NODE":local
|
||||
incus network delete "$NODE":incusbr0
|
||||
```
|
||||
|
||||
#### Step 4: Join nodes to the cluster
|
||||
|
||||
**Interactive** (one node at a time):
|
||||
|
||||
```bash
|
||||
# Generate join token
|
||||
incus cluster add incus-lab-01:incus-lab-02
|
||||
|
||||
# Step 3: Join second node
|
||||
# Join (answer the interactive prompts)
|
||||
incus cluster join incus-lab-01: incus-lab-02:
|
||||
# Prompt 1 - IP address: press Enter (accept default)
|
||||
# Prompt 2 - Member name: press Enter (accept default)
|
||||
# Prompt 3 - "All existing data is lost": type "yes"
|
||||
# Prompt 4 - source for storage pool "local": type "local/incus"
|
||||
# Prompt 5 - zfs.pool_name for storage pool "local": type "local/incus"
|
||||
```
|
||||
|
||||
# Step 4: Repeat for third node
|
||||
incus cluster add incus-lab-01: incus-lab-03
|
||||
incus cluster join incus-lab-01: incus-lab-03:
|
||||
**Automated** (non-interactive):
|
||||
|
||||
# Step 5: Verify
|
||||
```bash
|
||||
# Generate token + join in one go
|
||||
incus cluster add incus-lab-01:incus-lab-02
|
||||
printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join incus-lab-01: incus-lab-02:
|
||||
```
|
||||
|
||||
After each join, fix the remote (same certificate issue as the init node):
|
||||
|
||||
```bash
|
||||
incus remote remove incus-lab-02
|
||||
incus remote add incus-lab-02 https://<IP2>:8443 --accept-certificate
|
||||
```
|
||||
|
||||
Repeat for incus-lab-03.
|
||||
|
||||
#### Step 5: Verify the cluster
|
||||
|
||||
```bash
|
||||
# All 3 members should be ONLINE
|
||||
incus cluster list incus-lab-01:
|
||||
|
||||
# Storage pool should show on all members
|
||||
incus storage show incus-lab-01:local
|
||||
incus storage show incus-lab-01:local --target incus-lab-01
|
||||
incus storage show incus-lab-01:local --target incus-lab-02
|
||||
incus storage show incus-lab-01:local --target incus-lab-03
|
||||
|
||||
# All remotes should work
|
||||
incus info incus-lab-01: | head -3
|
||||
incus info incus-lab-02: | head -3
|
||||
incus info incus-lab-03: | head -3
|
||||
```
|
||||
|
||||
**Key points:**
|
||||
- `apply_defaults: true` on all nodes ensures matching storage pool and
|
||||
network bridge configurations (required for migration)
|
||||
network bridge names (required for migration) -- but the pools must be
|
||||
deleted on joining nodes before the join
|
||||
- After joining, manage the cluster through the init node's remote
|
||||
- The ZFS pool source is `local/incus` on IncusOS (auto-created by
|
||||
`apply_defaults`)
|
||||
- Each join generates a new token (tokens are single-use and time-limited)
|
||||
- VM migration requires matching storage pool names across nodes
|
||||
|
||||
### Migration Testing (After Cluster Formation)
|
||||
|
||||
Once the cluster is formed, validate container and VM migration:
|
||||
|
||||
#### Container migration (stop/move/start)
|
||||
|
||||
```bash
|
||||
# Launch on node 2
|
||||
incus launch images:debian/12 incus-lab-01:test-container --target incus-lab-02
|
||||
incus exec incus-lab-01:test-container -- bash -c 'echo "hello" > /root/test.txt'
|
||||
|
||||
# Migrate to node 3
|
||||
incus stop incus-lab-01:test-container
|
||||
incus move incus-lab-01:test-container --target incus-lab-03
|
||||
incus start incus-lab-01:test-container
|
||||
|
||||
# Verify: data survives, processes don't
|
||||
incus exec incus-lab-01:test-container -- cat /root/test.txt # "hello"
|
||||
incus list incus-lab-01: --columns nLs # node 3
|
||||
```
|
||||
|
||||
#### VM live migration
|
||||
|
||||
```bash
|
||||
# Launch and configure for migration
|
||||
incus launch images:debian/12 incus-lab-01:test-vm --vm --target incus-lab-02
|
||||
# Wait for VM agent...
|
||||
incus stop incus-lab-01:test-vm
|
||||
incus config set incus-lab-01:test-vm migration.stateful=true
|
||||
incus config set incus-lab-01:test-vm limits.cpu=0-1
|
||||
incus config device add incus-lab-01:test-vm root disk path=/ pool=local size.state=2GiB
|
||||
incus start incus-lab-01:test-vm
|
||||
|
||||
# Set up heartbeat for state verification
|
||||
incus exec incus-lab-01:test-vm -- mkdir -p /tmp/migration-test
|
||||
incus exec incus-lab-01:test-vm -- bash -c \
|
||||
'cat > /tmp/migration-test/heartbeat.sh << "EOF"
|
||||
#!/bin/bash
|
||||
i=0; while true; do echo $i > /tmp/migration-test/heartbeat; i=$((i+1)); sleep 1; done
|
||||
EOF
|
||||
chmod +x /tmp/migration-test/heartbeat.sh'
|
||||
incus exec incus-lab-01:test-vm -- \
|
||||
systemd-run --unit=migration-heartbeat /tmp/migration-test/heartbeat.sh
|
||||
|
||||
# Live-migrate (state preserved)
|
||||
sleep 3
|
||||
incus exec incus-lab-01:test-vm -- cat /tmp/migration-test/heartbeat # note value
|
||||
incus move incus-lab-01:test-vm --target incus-lab-03
|
||||
sleep 2 # agent needs ~1-2s to reconnect
|
||||
incus exec incus-lab-01:test-vm -- cat /tmp/migration-test/heartbeat # should be higher
|
||||
incus exec incus-lab-01:test-vm -- systemctl is-active migration-heartbeat # "active"
|
||||
```
|
||||
|
||||
#### Cluster evacuation
|
||||
|
||||
```bash
|
||||
# Move both workloads to node 2, then evacuate
|
||||
incus cluster evacuate incus-lab-01:incus-lab-02 --force
|
||||
incus cluster list incus-lab-01: # node 2 = EVACUATED
|
||||
incus list incus-lab-01: --columns nLs # workloads on other nodes
|
||||
|
||||
# Restore
|
||||
incus cluster restore incus-lab-01:incus-lab-02 --force
|
||||
incus cluster list incus-lab-01: # node 2 = ONLINE
|
||||
incus list incus-lab-01: --columns nLs # workloads back on node 2
|
||||
```
|
||||
|
||||
See `notes/clustering-guide.md` for the full automated migration test script
|
||||
and detailed test results.
|
||||
|
||||
### Cleanup
|
||||
|
||||
```bash
|
||||
|
|
@ -427,8 +634,12 @@ incus cluster list incus-lab-01:
|
|||
# Destroy VMs only
|
||||
./incusos/incusos-proxmox --cleanup examples/lab-cluster.yaml
|
||||
|
||||
# Full cleanup: VMs + ISOs + seeds + remotes + cache
|
||||
# Full cleanup: VMs + deployment ISO + seeds + remotes + cache
|
||||
./incusos/incusos-proxmox --cleanup --deep --yes examples/lab-cluster.yaml
|
||||
|
||||
# Pool-wide: destroy ALL managed VMs (no config file needed)
|
||||
./incusos/incusos-proxmox --cleanup-all --yes
|
||||
./incusos/incusos-proxmox --cleanup-all --deep --yes # nuke everything
|
||||
```
|
||||
|
||||
---
|
||||
|
|
@ -507,3 +718,7 @@ CLI is installed.
|
|||
| Pool not found | Resource pool not created | Run `pveum pool add <name>` on Proxmox |
|
||||
| Re-run destroys VMs | Used option 3 in reconcile | Use option 1 (checks) or option 2 (continue) instead |
|
||||
| `--status` shows "not found" | VM was cleaned up | Deploy first, then check status |
|
||||
| VM live migration fails: "Missing section footer for ICH9LPC" | `limits.cpu` not set or set as integer | Use range syntax: `limits.cpu=0-1` (not `2`); see `notes/clustering-guide.md` |
|
||||
| `migration.stateful` cannot be set while running | Incus requires VM to be stopped | Stop VM first, set config, then start |
|
||||
| `certificate is valid for 127.0.0.1, ::1, not <IP>` | Cluster cert regenerated on enable/join | Remove and re-add remote with `--accept-certificate` |
|
||||
| `Config key "zfs.pool_name" is cluster member specific` | Storage pool exists on joining node | Delete pool/network before joining; see clustering guide |
|
||||
|
|
|
|||
|
|
@ -0,0 +1,45 @@
|
|||
# lab-advanced.yaml - 3-node heterogeneous cluster for advanced migration testing
|
||||
#
|
||||
# Deploys 3 Incus nodes with a 6/4/4 core split to test VM live migration
|
||||
# across hosts with different CPU topologies. 8 GiB RAM provides headroom
|
||||
# for multiple concurrent VMs with size.state for stateful operations.
|
||||
#
|
||||
# Test matrix targets:
|
||||
# - Multi-vCPU VMs (2, 3, 4 vCPUs) with limits.cpu ranges
|
||||
# - Concurrent migrations from different source nodes
|
||||
# - Edge cases: active I/O, memory-loaded VMs, 4-vCPU on 4-core host
|
||||
# - Cluster lifecycle: evacuate, remove, replace, rejoin, rebalance
|
||||
#
|
||||
# Usage:
|
||||
# incusos-proxmox --dry-run examples/lab-advanced.yaml # Preview
|
||||
# incusos-proxmox --yes examples/lab-advanced.yaml # Deploy
|
||||
# incusos-proxmox --status examples/lab-advanced.yaml # Check status
|
||||
# incusos-proxmox --cleanup examples/lab-advanced.yaml # Tear down VMs
|
||||
# incusos-proxmox --cleanup --deep examples/lab-advanced.yaml # Full cleanup
|
||||
#
|
||||
# Connection settings from proxmox.yaml (see examples/proxmox.yaml.example).
|
||||
# Add a proxmox: section here to override per-deployment.
|
||||
#
|
||||
# Note: start_vmid 910 may collide with lab-oc.yaml. Clean up old deployments
|
||||
# first or adjust the range.
|
||||
|
||||
defaults:
|
||||
memory: 8192
|
||||
disk: 50
|
||||
start_vmid: 910
|
||||
|
||||
vms:
|
||||
- name: incus-adv-01
|
||||
app: incus
|
||||
apply_defaults: true
|
||||
cores: 6
|
||||
|
||||
- name: incus-adv-02
|
||||
app: incus
|
||||
apply_defaults: true
|
||||
cores: 4
|
||||
|
||||
- name: incus-adv-03
|
||||
app: incus
|
||||
apply_defaults: true
|
||||
cores: 4
|
||||
|
|
@ -13,16 +13,12 @@
|
|||
#
|
||||
# After deployment, form a cluster with lab-test:
|
||||
# lab-test --phase cluster examples/lab-cluster.yaml
|
||||
|
||||
proxmox:
|
||||
host: 192.168.1.29
|
||||
method: api
|
||||
api_token_id: automation@pve!deploy
|
||||
storage: local-zfs
|
||||
pool: IncusLab
|
||||
#
|
||||
# Connection settings from proxmox.yaml (see examples/proxmox.yaml.example).
|
||||
# Add a proxmox: section here to override per-deployment.
|
||||
|
||||
defaults:
|
||||
cores: 2
|
||||
cores: 4
|
||||
memory: 4096
|
||||
disk: 50
|
||||
start_vmid: 900
|
||||
|
|
@ -31,9 +27,6 @@ vms:
|
|||
- name: incus-lab-01
|
||||
app: incus
|
||||
apply_defaults: true
|
||||
cores: 4
|
||||
memory: 8192
|
||||
disk: 64
|
||||
|
||||
- name: incus-lab-02
|
||||
app: incus
|
||||
|
|
|
|||
|
|
@ -15,13 +15,9 @@
|
|||
# incusos-proxmox --yes examples/lab-oc.yaml
|
||||
# incusos-proxmox --status examples/lab-oc.yaml
|
||||
# incusos-proxmox --cleanup --deep examples/lab-oc.yaml
|
||||
|
||||
proxmox:
|
||||
host: 192.168.1.29
|
||||
method: api
|
||||
api_token_id: automation@pve!deploy
|
||||
storage: local-zfs
|
||||
pool: IncusLab
|
||||
#
|
||||
# Connection settings from proxmox.yaml (see examples/proxmox.yaml.example).
|
||||
# Add a proxmox: section here to override per-deployment.
|
||||
|
||||
defaults:
|
||||
cores: 2
|
||||
|
|
|
|||
|
|
@ -0,0 +1,24 @@
|
|||
# lab-replace.yaml - Single replacement node for cluster lifecycle testing
|
||||
#
|
||||
# Used in Part 4 of the advanced migration test plan: after evacuating,
|
||||
# removing, and destroying incus-adv-03, this config deploys a fresh
|
||||
# replacement node that gets joined back into the cluster.
|
||||
#
|
||||
# Usage:
|
||||
# incusos-proxmox --dry-run examples/lab-replace.yaml
|
||||
# incusos-proxmox --yes examples/lab-replace.yaml
|
||||
# incusos-proxmox --status examples/lab-replace.yaml
|
||||
# incusos-proxmox --cleanup examples/lab-replace.yaml
|
||||
#
|
||||
# Connection settings from proxmox.yaml (see examples/proxmox.yaml.example).
|
||||
|
||||
defaults:
|
||||
memory: 8192
|
||||
disk: 50
|
||||
start_vmid: 913
|
||||
|
||||
vms:
|
||||
- name: incus-adv-04
|
||||
app: incus
|
||||
apply_defaults: true
|
||||
cores: 4
|
||||
|
|
@ -1,9 +1,5 @@
|
|||
proxmox:
|
||||
host: 192.168.1.29
|
||||
method: api
|
||||
api_token_id: automation@pve!deploy
|
||||
storage: local-zfs # VM disk storage (default: local-lvm)
|
||||
pool: IncusLab
|
||||
# Connection settings from proxmox.yaml (see examples/proxmox.yaml.example).
|
||||
# Add a proxmox: section here to override per-deployment.
|
||||
|
||||
defaults:
|
||||
cores: 2
|
||||
|
|
|
|||
|
|
@ -0,0 +1,21 @@
|
|||
# proxmox.yaml.example - Proxmox connection config template
|
||||
#
|
||||
# Copy to incusos/proxmox.yaml and edit with your settings.
|
||||
# This file is auto-discovered by incusos-proxmox and lab-test.
|
||||
#
|
||||
# Lab config files (lab-cluster.yaml, lab-oc.yaml, etc.) reference
|
||||
# this for connection settings, so you only need to change credentials
|
||||
# in one place.
|
||||
#
|
||||
# Override any setting per-deployment by adding a proxmox: section
|
||||
# to the lab config file.
|
||||
|
||||
host: 192.168.1.10 # Proxmox host IP or hostname
|
||||
method: ssh # ssh (default) or api
|
||||
# api_token_id: automation@pve!deploy # For api method
|
||||
node: pve # Proxmox node name
|
||||
storage: local-lvm # VM disk storage
|
||||
iso_storage: local # ISO/image storage
|
||||
bridge: vmbr0 # Network bridge
|
||||
# pool: IncusLab # Resource pool (optional, for isolation)
|
||||
ssh_user: root # SSH user (for ssh method)
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
Loading…
Reference in New Issue