Add advanced migration testing, cluster lifecycle, and clustering guide
- New lab-advanced.yaml (3-node 6/4/4 core heterogeneous cluster) and lab-replace.yaml (single replacement node) for advanced migration and cluster lifecycle testing - New notes/clustering-guide.md with full clustering walkthrough: cluster formation, storage pool conflict fix, join workflow, VM live migration with limits.cpu range fix, multi-vCPU test results (2/3/4 vCPUs), edge cases (I/O, memory, concurrent), and node replacement lifecycle - Updated CLAUDE.md with clustering docs, rebalancing config keys, cluster enable syntax fix (TWO args not one), cluster remove syntax, agent reconnect timing (3-4s not 1-2s), and lifecycle procedure - proxmox.yaml.example for Proxmox connection config reference Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
2aca6fb33b
commit
732d70b38e
|
|
@ -25,3 +25,6 @@ Thumbs.db
|
||||||
|
|
||||||
# Go binaries (flasher-tool)
|
# Go binaries (flasher-tool)
|
||||||
flasher-tool
|
flasher-tool
|
||||||
|
|
||||||
|
# Proxmox connection config (contains credentials)
|
||||||
|
proxmox.yaml
|
||||||
|
|
|
||||||
226
CLAUDE.md
226
CLAUDE.md
|
|
@ -19,9 +19,11 @@ incu-contrib/
|
||||||
│ ├── incusos-seed # Seed archive generator
|
│ ├── incusos-seed # Seed archive generator
|
||||||
│ ├── incusos-proxmox # Declarative Proxmox VM deployment
|
│ ├── incusos-proxmox # Declarative Proxmox VM deployment
|
||||||
│ ├── lab-test # Guided lab validation (workloads, clustering, migration)
|
│ ├── lab-test # Guided lab validation (workloads, clustering, migration)
|
||||||
|
│ ├── proxmox.yaml # Proxmox connection config (gitignored, contains credentials)
|
||||||
│ ├── TESTING.md # Testing guide for incusos-proxmox and lab-test
|
│ ├── TESTING.md # Testing guide for incusos-proxmox and lab-test
|
||||||
│ └── examples/ # Example seed + Proxmox YAML files
|
│ └── examples/ # Example seed + Proxmox YAML files
|
||||||
└── notes/ # Research notes and reference material
|
└── notes/ # Research notes and reference material
|
||||||
|
└── clustering-guide.md # Detailed Incus clustering walkthrough
|
||||||
```
|
```
|
||||||
|
|
||||||
## Key technical context
|
## Key technical context
|
||||||
|
|
@ -78,6 +80,11 @@ incu-contrib/
|
||||||
- **`incusos-proxmox`** reads a YAML config, generates per-VM SEED_DATA images
|
- **`incusos-proxmox`** reads a YAML config, generates per-VM SEED_DATA images
|
||||||
via `incusos-seed --format iso`, uploads the ISO + seeds to Proxmox, creates
|
via `incusos-seed --format iso`, uploads the ISO + seeds to Proxmox, creates
|
||||||
VMs with IncusOS-correct settings, and boots them through installation.
|
VMs with IncusOS-correct settings, and boots them through installation.
|
||||||
|
- **Config separation**: Proxmox connection settings (host, credentials, pool)
|
||||||
|
live in `incusos/proxmox.yaml` (gitignored). Lab configs (`lab-cluster.yaml`,
|
||||||
|
etc.) only define VM specs. Merge priority: `proxmox.yaml` base → lab config
|
||||||
|
`proxmox:` overlay → CLI flags (`--host`, `--method`). Auto-discovery looks
|
||||||
|
for `proxmox.yaml` in script directory then cwd; override with `--proxmox FILE`.
|
||||||
- **Connection methods**: SSH (default, `ssh root@host qm ...`) or API
|
- **Connection methods**: SSH (default, `ssh root@host qm ...`) or API
|
||||||
(`curl -k https://host:8006/api2/json/...` with `PVEAPIToken` header).
|
(`curl -k https://host:8006/api2/json/...` with `PVEAPIToken` header).
|
||||||
- **Minimum API privileges** for token-based access:
|
- **Minimum API privileges** for token-based access:
|
||||||
|
|
@ -142,27 +149,198 @@ incu-contrib/
|
||||||
|
|
||||||
- **Cluster formation** is done entirely through the `incus` CLI using remotes.
|
- **Cluster formation** is done entirely through the `incus` CLI using remotes.
|
||||||
No SSH to the IncusOS nodes is needed (IncusOS is immutable, no shell access).
|
No SSH to the IncusOS nodes is needed (IncusOS is immutable, no shell access).
|
||||||
- **Workflow:**
|
- **No VIP needed**: each node advertises its own IP as its cluster address.
|
||||||
|
Clients can connect to any cluster member; requests are forwarded internally.
|
||||||
|
|
||||||
|
#### Pre-clustering: fix core.https_address
|
||||||
|
|
||||||
|
- IncusOS nodes default to `core.https_address: :8443` (wildcard / all
|
||||||
|
interfaces). Clustering requires a **specific routable IP** so nodes can
|
||||||
|
address each other.
|
||||||
|
- **Set the IP on every node BEFORE enabling clustering:**
|
||||||
```bash
|
```bash
|
||||||
# Enable clustering on first node
|
incus config set <remote>: core.https_address <NODE_IP>:8443
|
||||||
incus cluster enable <remote1>: <member-name>
|
|
||||||
|
|
||||||
# Generate join token (on cluster, for new member name)
|
|
||||||
incus cluster add <remote1>: <new-member-name>
|
|
||||||
|
|
||||||
# Join second node to cluster
|
|
||||||
incus cluster join <remote1>: <remote2>:
|
|
||||||
|
|
||||||
# Verify
|
|
||||||
incus cluster list <remote1>:
|
|
||||||
```
|
```
|
||||||
- **`apply_defaults: true`** must be set on all nodes that will be clustered --
|
- Get each node's routable IP via the API:
|
||||||
this ensures matching storage pool (`local`) and network bridge names, which
|
```bash
|
||||||
are required for migration.
|
incus query <remote>:/1.0 | python3 -c "import sys,json; d=json.load(sys.stdin); \
|
||||||
|
[print(a) for a in d['environment']['addresses'] \
|
||||||
|
if not a.startswith('10.') and not a.startswith('fd42:') and not a.startswith('[')]"
|
||||||
|
```
|
||||||
|
- This is safe to do while remotes are connected -- the remote already points
|
||||||
|
to the specific IP; we're just narrowing the bind address. Certificate trust
|
||||||
|
is fingerprint-based, not address-based.
|
||||||
|
|
||||||
|
#### Cluster enable (init node)
|
||||||
|
|
||||||
|
- ```bash
|
||||||
|
incus cluster enable <remote>: <member-name>
|
||||||
|
```
|
||||||
|
Note: this is TWO arguments: `<remote>:` (trailing colon) and `<member-name>`.
|
||||||
|
The help text shows `[<remote>:] <name>` — NOT `remote:name` as a single arg.
|
||||||
|
- **TLS certificate regeneration**: enabling clustering causes the server to
|
||||||
|
generate a new TLS certificate (cluster cert). The new cert may only have
|
||||||
|
SANs for `127.0.0.1` and `::1`, breaking the existing remote.
|
||||||
|
- **Fix**: remove and re-add the remote to pin the new certificate:
|
||||||
|
```bash
|
||||||
|
incus remote switch local # if init remote is current default
|
||||||
|
incus remote remove <remote>
|
||||||
|
incus remote add <remote> https://<NODE_IP>:8443 --accept-certificate
|
||||||
|
```
|
||||||
|
- The cert trust on the server side (client → server) is unaffected -- it's
|
||||||
|
stored by fingerprint in the Incus database, independent of listen address.
|
||||||
|
|
||||||
|
#### Joining nodes: the apply_defaults storage pool conflict
|
||||||
|
|
||||||
|
- **`apply_defaults: true`** creates a `local` ZFS storage pool and `incusbr0`
|
||||||
|
network bridge on each node at install time. When joining a cluster that
|
||||||
|
already has a `local` pool defined, the join wizard asks for `source` and
|
||||||
|
`zfs.pool_name` properties but then tries to set them as cluster-wide config.
|
||||||
|
Incus rejects this because they are **member-specific** keys.
|
||||||
|
- **The fix**: delete the storage pool and network on the joining node before
|
||||||
|
joining. The join process then creates the member-specific pool entry
|
||||||
|
correctly.
|
||||||
|
```bash
|
||||||
|
# 1. Remove config references
|
||||||
|
incus config unset <remote>: storage.backups_volume
|
||||||
|
incus config unset <remote>: storage.images_volume
|
||||||
|
# 2. Delete volumes
|
||||||
|
incus storage volume delete <remote>:local backups
|
||||||
|
incus storage volume delete <remote>:local images
|
||||||
|
# 3. Clear default profile references (pool is "in use" otherwise)
|
||||||
|
incus profile device remove <remote>:default root
|
||||||
|
incus profile device remove <remote>:default eth0
|
||||||
|
# 4. Delete pool and network
|
||||||
|
incus storage delete <remote>:local
|
||||||
|
incus network delete <remote>:incusbr0
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Join workflow
|
||||||
|
|
||||||
|
- **Generate token** (on init node, single argument `remote:member-name`):
|
||||||
|
```bash
|
||||||
|
incus cluster add <init-remote>:<new-member-name>
|
||||||
|
```
|
||||||
|
- **Join** (interactive -- prompts for 5 values):
|
||||||
|
```bash
|
||||||
|
incus cluster join <init-remote>: <joining-remote>:
|
||||||
|
```
|
||||||
|
Interactive prompts and correct answers:
|
||||||
|
1. IP address → accept default (node's IP, already set via core.https_address)
|
||||||
|
2. Member name → accept default (matches the token)
|
||||||
|
3. "All existing data is lost" → `yes`
|
||||||
|
4. `source` property for storage pool "local" → `local/incus`
|
||||||
|
5. `zfs.pool_name` property for storage pool "local" → `local/incus`
|
||||||
|
- **Automated (non-interactive):**
|
||||||
|
```bash
|
||||||
|
printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join <init-remote>: <joining-remote>:
|
||||||
|
```
|
||||||
|
- **After join**: the joining node gets a new cluster certificate. Fix the
|
||||||
|
remote (same as init node):
|
||||||
|
```bash
|
||||||
|
incus remote remove <joining-remote>
|
||||||
|
incus remote add <joining-remote> https://<NODE_IP>:8443 --accept-certificate
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Command syntax gotchas
|
||||||
|
|
||||||
|
- `incus cluster enable remote: member-name` -- TWO arguments (remote: + name)
|
||||||
|
- `incus cluster add remote:member-name` -- ONE argument (no space)
|
||||||
|
- `incus cluster remove remote:member-name --force` -- ONE argument; prompts
|
||||||
|
"yes/no" even with `--force`, pipe `printf "yes\n"` for automation
|
||||||
|
- `incus cluster evacuate remote:member-name` -- ONE argument (no space)
|
||||||
|
- `incus cluster restore remote:member-name` -- ONE argument (no space)
|
||||||
|
- `incus cluster join init-remote: joining-remote:` -- TWO arguments (space)
|
||||||
|
- `incus storage show remote:pool` -- ONE argument (no space)
|
||||||
|
- `incus storage show remote:pool --target member` -- target flag for
|
||||||
|
member-specific config
|
||||||
|
- `incus config set remote: key value` -- remote with trailing colon + space
|
||||||
|
- General rule: `remote:resource` for targeting a resource, `remote:` (trailing
|
||||||
|
colon) for targeting the server itself
|
||||||
|
|
||||||
|
#### Post-join state
|
||||||
|
|
||||||
- After joining, the cluster is managed through the init node's remote. The
|
- After joining, the cluster is managed through the init node's remote. The
|
||||||
individual node remotes still work for node-specific operations.
|
individual node remotes still work for node-specific operations.
|
||||||
- **`lab-test`** automates cluster formation, workload testing, and migration.
|
- **`lab-test`** automates cluster formation, workload testing, and migration.
|
||||||
|
|
||||||
|
#### Workload placement and migration
|
||||||
|
|
||||||
|
- **Targeted launch**: `incus launch images:debian/12 <cluster-remote>:name --target <member>`
|
||||||
|
- **Cluster-wide visibility**: `incus list` on any member shows all instances.
|
||||||
|
- **Container migration**: stop/move/start only (CRIU live migration is
|
||||||
|
unreliable). Data persists, processes do not.
|
||||||
|
```bash
|
||||||
|
incus stop <remote>:<instance>
|
||||||
|
incus move <remote>:<instance> --target <destination-member>
|
||||||
|
incus start <remote>:<instance>
|
||||||
|
```
|
||||||
|
- **VM live migration**: requires `migration.stateful=true` (must be set
|
||||||
|
while VM is stopped). Preserves running state with no downtime.
|
||||||
|
```bash
|
||||||
|
incus move <remote>:<instance> --target <destination-member>
|
||||||
|
```
|
||||||
|
- **VM live migration requires `limits.cpu` as a range** (e.g., `0-1`),
|
||||||
|
not an integer. Without this, Incus sets QEMU's `maxcpus` to the host's
|
||||||
|
CPU count (`driver_qemu_templates.go`: `maxcpus = min(cpu.Total, 64)`).
|
||||||
|
Different `maxcpus` values size the ICH9 ACPI CPU hotplug state arrays
|
||||||
|
differently, causing `Missing section footer for ICH9LPC` on restore.
|
||||||
|
Using a range (pinning syntax) eliminates `maxcpus` entirely and uses
|
||||||
|
fixed `sockets/cores/threads` topology — portable across all hosts.
|
||||||
|
```bash
|
||||||
|
# WRONG: integer → maxcpus varies by host → migration fails
|
||||||
|
incus config set <instance> limits.cpu=2
|
||||||
|
# RIGHT: range → fixed topology → migration works everywhere
|
||||||
|
incus config set <instance> limits.cpu=0-1
|
||||||
|
```
|
||||||
|
- **VM live migration works in nested virtualization** (IncusOS inside
|
||||||
|
Proxmox on Intel). It is NOT limited to bare metal. Tested with QEMU
|
||||||
|
10.2.1 on Intel i9-13900HK with heterogeneous host core counts (4 vs 2).
|
||||||
|
- **The `vnmi` CPUID warning** (`CPUID[eax=8000000Ah].EDX.vnmi`) that
|
||||||
|
appears during migration is cosmetic. It fires from QEMU's feature
|
||||||
|
dependency checker before KVM filters out unsupported features and does
|
||||||
|
not affect migration.
|
||||||
|
- **Stateful stop/restore** (`incus stop --stateful` + `incus start`) also
|
||||||
|
requires the `limits.cpu` range fix. Use `incus start --stateless` to
|
||||||
|
discard a saved state file that cannot be restored.
|
||||||
|
- **VM `size.state` config**: stateful operations require `size.state` on
|
||||||
|
the root disk (`incus config device add <instance> root disk path=/
|
||||||
|
pool=local size.state=2GiB`). Without it, `incus stop --stateful` fails.
|
||||||
|
- **Cluster evacuation**: `incus cluster evacuate <remote>:<member> --force`
|
||||||
|
(ONE argument, like `cluster enable` and `cluster add`).
|
||||||
|
Use `--action stop` if VMs lack the `limits.cpu` range fix.
|
||||||
|
Restore with `incus cluster restore <remote>:<member> --force`.
|
||||||
|
- **VM agent reconnect**: after live migration, the incus agent inside the VM
|
||||||
|
needs ~3-4 seconds to reconnect. `incus exec` commands issued immediately
|
||||||
|
after migration may fail with "VM agent isn't currently running". Scripts
|
||||||
|
should `sleep 4` after migration before running `incus exec`.
|
||||||
|
- **Multi-vCPU migration**: tested with 2, 3, and 4 vCPU VMs across
|
||||||
|
heterogeneous hosts (6/4/4 cores). Odd vCPU counts (e.g., `limits.cpu=0-2`)
|
||||||
|
work identically to even counts. A 4-vCPU VM on a 4-core host (100% core
|
||||||
|
usage) migrates without issues. `size.state=4GiB` recommended for 3-4 vCPU VMs.
|
||||||
|
- **Concurrent migrations**: migrating multiple VMs simultaneously from
|
||||||
|
different source nodes works without interference. ~140 MB/s per migration.
|
||||||
|
- **Active I/O during migration**: disk writes and network activity survive
|
||||||
|
live migration transparently. File integrity verified after migration.
|
||||||
|
- **Cluster rebalancing**: Incus can auto-redistribute VMs when a new node
|
||||||
|
joins. Only moves VMs with `migration.stateful=true`. Containers are NOT
|
||||||
|
auto-rebalanced.
|
||||||
|
```bash
|
||||||
|
incus config set <remote>: cluster.rebalance.interval=1 # minutes
|
||||||
|
incus config set <remote>: cluster.rebalance.threshold=10 # imbalance %
|
||||||
|
incus config set <remote>: cluster.rebalance.batch=2 # max VMs/run
|
||||||
|
incus config set <remote>: cluster.rebalance.cooldown=5m # wait between runs
|
||||||
|
```
|
||||||
|
- **Node replacement lifecycle**: evacuate → remove → destroy → deploy fresh →
|
||||||
|
join → auto-rebalance. Full procedure tested. See `notes/clustering-guide.md`
|
||||||
|
for step-by-step instructions.
|
||||||
|
- **`incus cluster remove` requires confirmation**: even with `--force`, it
|
||||||
|
prompts "Are you really sure?". Pipe `yes` for automation:
|
||||||
|
```bash
|
||||||
|
printf "yes\n" | incus cluster remove <remote>:<member> --force
|
||||||
|
```
|
||||||
|
- See `notes/clustering-guide.md` for full details and references.
|
||||||
|
|
||||||
### Lab validation (lab-test)
|
### Lab validation (lab-test)
|
||||||
|
|
||||||
- **`lab-test`** reads the same YAML config as `incusos-proxmox` and operates
|
- **`lab-test`** reads the same YAML config as `incusos-proxmox` and operates
|
||||||
|
|
@ -187,11 +365,19 @@ incu-contrib/
|
||||||
### incusos-proxmox doctor and cleanup
|
### incusos-proxmox doctor and cleanup
|
||||||
|
|
||||||
- **`--doctor`**: standalone environment check. No config file required.
|
- **`--doctor`**: standalone environment check. No config file required.
|
||||||
Checks tool versions, IncusOS CDN, and optionally Proxmox connectivity.
|
Checks tool versions, IncusOS CDN, proxmox.yaml discovery, and optionally
|
||||||
- **`--cleanup --deep`**: destroys VMs + deletes ISOs/seeds from Proxmox +
|
Proxmox connectivity (from proxmox.yaml or config file).
|
||||||
removes incus remotes + clears local ISO cache.
|
- **`--cleanup`**: destroys VMs defined in the config file.
|
||||||
- **`--cleanup-all`**: pool-wide cleanup. Destroys all VMs with the
|
- **`--cleanup --deep`**: also deletes the specific IncusOS ISO used by this
|
||||||
`[incusos-lab:managed]` marker in the configured resource pool.
|
deployment + per-VM seed ISOs + incus remotes + local cache. Does NOT delete
|
||||||
|
all IncusOS ISOs (unlike the old behavior).
|
||||||
|
- **`--cleanup-all`**: pool-wide cleanup. Only needs `proxmox.yaml` (no lab
|
||||||
|
config required). Destroys all VMs with `[incusos-lab:managed]` marker.
|
||||||
|
- **`--cleanup-all --deep`**: aggressive blanket delete of ALL `IncusOS_*.iso`
|
||||||
|
and `seed-*.iso` from storage + remotes + cache.
|
||||||
|
- **`--verbose` / `-v`**: shows detailed output (tool paths, API calls). Default
|
||||||
|
output is concise (step names + results). `--quiet` suppresses everything
|
||||||
|
except warnings and errors.
|
||||||
|
|
||||||
## Coding conventions for scripts
|
## Coding conventions for scripts
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -296,13 +296,16 @@ This script ensures all settings are correct, preventing install failures.
|
||||||
--no-cert Skip certificate injection
|
--no-cert Skip certificate injection
|
||||||
--vm NAME Operate on a single VM only
|
--vm NAME Operate on a single VM only
|
||||||
-y, --yes Skip confirmation prompts
|
-y, --yes Skip confirmation prompts
|
||||||
|
--proxmox FILE Proxmox connection config (default: auto-discover)
|
||||||
--doctor Check tool versions and environment health
|
--doctor Check tool versions and environment health
|
||||||
--status Check deployment status (shortcut for --phase status)
|
--status Check deployment status (shortcut for --phase status)
|
||||||
--cleanup Destroy VMs defined in config
|
--cleanup Destroy VMs defined in config
|
||||||
--cleanup --deep Also remove ISOs, seeds, remotes, and local cache
|
--cleanup --deep Also remove deployment ISO, seeds, remotes, and cache
|
||||||
--cleanup-all Destroy ALL managed VMs in the resource pool
|
--cleanup-all Destroy ALL managed VMs in pool (no config file needed)
|
||||||
|
--cleanup-all --deep Also remove ALL ISOs and seeds from storage
|
||||||
--force-cleanup Destroy VMs even without management marker
|
--force-cleanup Destroy VMs even without management marker
|
||||||
--dry-run Preview without executing
|
--dry-run Preview without executing
|
||||||
|
-v, --verbose Show detailed output (tool paths, API calls)
|
||||||
-q, --quiet Suppress info output
|
-q, --quiet Suppress info output
|
||||||
-h, --help Full help text
|
-h, --help Full help text
|
||||||
-V, --version Show version
|
-V, --version Show version
|
||||||
|
|
@ -314,14 +317,15 @@ Run `--doctor` to verify your environment without a config file:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
./incusos-proxmox --doctor
|
./incusos-proxmox --doctor
|
||||||
|
./incusos-proxmox --doctor -v # verbose: show tool paths
|
||||||
```
|
```
|
||||||
|
|
||||||
This checks: incus client, flasher-tool, operations-center CLI (optional),
|
This checks: incus client, flasher-tool, operations-center CLI (optional),
|
||||||
python3, curl, genisoimage, jq, incusos-seed sibling, and IncusOS CDN
|
python3, curl, genisoimage, jq, incusos-seed sibling, IncusOS CDN
|
||||||
availability.
|
availability, and `proxmox.yaml` discovery.
|
||||||
|
|
||||||
If a config file is also provided, additionally checks Proxmox connectivity,
|
If `proxmox.yaml` is found or a config file is provided, additionally checks
|
||||||
pool existence, and API token validity:
|
Proxmox connectivity, pool existence, and API token validity:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
./incusos-proxmox --doctor examples/lab-cluster.yaml
|
./incusos-proxmox --doctor examples/lab-cluster.yaml
|
||||||
|
|
@ -329,22 +333,28 @@ pool existence, and API token validity:
|
||||||
|
|
||||||
### Deep Cleanup
|
### Deep Cleanup
|
||||||
|
|
||||||
Use `--deep` with `--cleanup` or `--cleanup-all` to remove all artifacts:
|
Use `--deep` with `--cleanup` or `--cleanup-all` to remove artifacts beyond VMs:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# VMs + ISOs + seeds + incus remotes + local cache
|
# VMs + the IncusOS ISO used by this deployment + seeds + remotes + cache
|
||||||
./incusos-proxmox --cleanup --deep examples/lab-cluster.yaml
|
./incusos-proxmox --cleanup --deep examples/lab-cluster.yaml
|
||||||
|
|
||||||
# Pool-wide: destroy all managed VMs + artifacts
|
# Pool-wide: destroy all managed VMs (uses proxmox.yaml, no config needed)
|
||||||
./incusos-proxmox --cleanup-all --deep examples/lab-cluster.yaml
|
./incusos-proxmox --cleanup-all
|
||||||
|
./incusos-proxmox --cleanup-all --deep # also nuke ALL ISOs + seeds
|
||||||
```
|
```
|
||||||
|
|
||||||
`--deep` additionally:
|
`--cleanup --deep`:
|
||||||
- Deletes seed ISOs from Proxmox storage
|
- Deletes per-VM seed ISOs from Proxmox storage
|
||||||
- Deletes IncusOS ISOs from Proxmox storage
|
- Deletes the specific IncusOS ISO used by this deployment (not all of them)
|
||||||
- Removes matching `incus remote` entries
|
- Removes matching `incus remote` entries
|
||||||
- Clears the local ISO cache (`~/.cache/incusos-proxmox/iso-cache/`)
|
- Clears the local ISO cache (`~/.cache/incusos-proxmox/iso-cache/`)
|
||||||
|
|
||||||
|
`--cleanup-all --deep`:
|
||||||
|
- Deletes ALL `IncusOS_*.iso` and `seed-*.iso` from storage (intentionally aggressive)
|
||||||
|
- Removes remotes for the destroyed VMs
|
||||||
|
- Clears the local ISO cache
|
||||||
|
|
||||||
### Deployment Phases
|
### Deployment Phases
|
||||||
|
|
||||||
| Phase | What it does |
|
| Phase | What it does |
|
||||||
|
|
@ -356,18 +366,41 @@ Use `--deep` with `--cleanup` or `--cleanup-all` to remove all artifacts:
|
||||||
| `install` | Start VMs, detect install completion via disk I/O monitoring, detach CD-ROMs, boot from disk |
|
| `install` | Start VMs, detect install completion via disk I/O monitoring, detach CD-ROMs, boot from disk |
|
||||||
| `status` | Check deployment status: Proxmox state, IP, port 8443, incus remotes, post-deploy checks |
|
| `status` | Check deployment status: Proxmox state, IP, port 8443, incus remotes, post-deploy checks |
|
||||||
|
|
||||||
### Config File Format
|
### Proxmox Connection Config
|
||||||
|
|
||||||
|
Proxmox connection settings live in a separate `proxmox.yaml` file to avoid
|
||||||
|
duplicating credentials across lab configs. Auto-discovered in the script
|
||||||
|
directory and current directory. Override with `--proxmox FILE`.
|
||||||
|
|
||||||
|
Copy the template to get started:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cp incusos/examples/proxmox.yaml.example incusos/proxmox.yaml
|
||||||
|
# Edit with your Proxmox host, credentials, and storage settings
|
||||||
|
```
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
|
# proxmox.yaml (flat structure, no wrapping key)
|
||||||
|
host: 192.168.1.10
|
||||||
|
method: api
|
||||||
|
api_token_id: automation@pve!deploy
|
||||||
|
node: pve
|
||||||
|
storage: local-lvm
|
||||||
|
iso_storage: local
|
||||||
|
bridge: vmbr0
|
||||||
|
pool: IncusLab
|
||||||
|
ssh_user: root
|
||||||
|
```
|
||||||
|
|
||||||
|
Lab configs can override any setting by including a `proxmox:` section.
|
||||||
|
Merge priority: `proxmox.yaml` → lab config `proxmox:` → CLI flags.
|
||||||
|
|
||||||
|
### Lab Config File Format
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
# proxmox: section is optional — overrides proxmox.yaml if present
|
||||||
proxmox:
|
proxmox:
|
||||||
host: 192.168.1.10 # Proxmox host (required)
|
storage: local-zfs # Override storage for this deployment
|
||||||
node: pve # Node name (default: pve)
|
|
||||||
storage: local-lvm # VM disk storage (default: local-lvm)
|
|
||||||
iso_storage: local # ISO storage (default: local)
|
|
||||||
bridge: vmbr0 # Network bridge (default: vmbr0)
|
|
||||||
pool: IncusLab # Resource pool (optional, for isolation)
|
|
||||||
method: ssh # ssh (default) or api
|
|
||||||
ssh_user: root # For ssh method
|
|
||||||
|
|
||||||
defaults:
|
defaults:
|
||||||
cores: 4
|
cores: 4
|
||||||
|
|
@ -602,10 +635,11 @@ See the [`examples/`](examples/) directory for sample seed configurations:
|
||||||
- [`incus-cluster-node.yaml`](examples/incus-cluster-node.yaml) -- Cluster node (no defaults)
|
- [`incus-cluster-node.yaml`](examples/incus-cluster-node.yaml) -- Cluster node (no defaults)
|
||||||
- [`ops-center.yaml`](examples/ops-center.yaml) -- Operations Center
|
- [`ops-center.yaml`](examples/ops-center.yaml) -- Operations Center
|
||||||
- [`network-static.yaml`](examples/network-static.yaml) -- Static network configuration
|
- [`network-static.yaml`](examples/network-static.yaml) -- Static network configuration
|
||||||
- [`proxmox-lab.yaml`](examples/proxmox-lab.yaml) -- Proxmox: 4-VM lab (ops-center + 3-node cluster)
|
- [`proxmox.yaml.example`](examples/proxmox.yaml.example) -- Proxmox connection config template
|
||||||
- [`proxmox-minimal.yaml`](examples/proxmox-minimal.yaml) -- Proxmox: single standalone VM
|
- [`proxmox-lab.yaml`](examples/proxmox-lab.yaml) -- Proxmox: 4-VM lab (self-contained, includes proxmox: section)
|
||||||
- [`lab-cluster.yaml`](examples/lab-cluster.yaml) -- Proxmox: 3-node cluster for lab validation
|
- [`proxmox-minimal.yaml`](examples/proxmox-minimal.yaml) -- Proxmox: single standalone VM (self-contained)
|
||||||
- [`lab-oc.yaml`](examples/lab-oc.yaml) -- Proxmox: Operations Center + 3 nodes
|
- [`lab-cluster.yaml`](examples/lab-cluster.yaml) -- Proxmox: 3-node cluster (uses proxmox.yaml)
|
||||||
|
- [`lab-oc.yaml`](examples/lab-oc.yaml) -- Proxmox: Operations Center + 3 nodes (uses proxmox.yaml)
|
||||||
|
|
||||||
## Security Notes
|
## Security Notes
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -129,13 +129,23 @@ ssh root@<YOUR_PROXMOX_IP> pvesh get /version
|
||||||
You should see JSON with the Proxmox VE version. If this fails, fix SSH
|
You should see JSON with the Proxmox VE version. If this fails, fix SSH
|
||||||
key auth first (`ssh-copy-id root@<host>`).
|
key auth first (`ssh-copy-id root@<host>`).
|
||||||
|
|
||||||
### Step 2: Create a Test Config
|
### Step 2: Set Up Proxmox Connection
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cp incusos/examples/proxmox.yaml.example incusos/proxmox.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
Edit `incusos/proxmox.yaml` with your Proxmox host settings. This is
|
||||||
|
auto-discovered by the scripts and avoids repeating credentials in each
|
||||||
|
lab config.
|
||||||
|
|
||||||
|
### Step 3: Create a Test Config
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cp incusos/examples/proxmox-minimal.yaml my-test.yaml
|
cp incusos/examples/proxmox-minimal.yaml my-test.yaml
|
||||||
```
|
```
|
||||||
|
|
||||||
Edit `my-test.yaml` -- change only the host IP:
|
Edit `my-test.yaml` -- change only the host IP (or rely on proxmox.yaml):
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
proxmox:
|
proxmox:
|
||||||
|
|
@ -155,7 +165,7 @@ vms:
|
||||||
|
|
||||||
Using VMID 900+ avoids clashing with existing VMs.
|
Using VMID 900+ avoids clashing with existing VMs.
|
||||||
|
|
||||||
### Step 3: Dry Run
|
### Step 4: Dry Run
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
./incusos/incusos-proxmox --dry-run my-test.yaml
|
./incusos/incusos-proxmox --dry-run my-test.yaml
|
||||||
|
|
@ -172,7 +182,7 @@ Using VMID 900+ avoids clashing with existing VMs.
|
||||||
- `[dry-run]` prefix on all actions -- nothing actually executed
|
- `[dry-run]` prefix on all actions -- nothing actually executed
|
||||||
- No errors
|
- No errors
|
||||||
|
|
||||||
### Step 4: Deploy (Single VM)
|
### Step 5: Deploy (Single VM)
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
./incusos/incusos-proxmox my-test.yaml
|
./incusos/incusos-proxmox my-test.yaml
|
||||||
|
|
@ -196,7 +206,7 @@ Using VMID 900+ avoids clashing with existing VMs.
|
||||||
disk, and reboot
|
disk, and reboot
|
||||||
- After reboot you should see the IncusOS login prompt
|
- After reboot you should see the IncusOS login prompt
|
||||||
|
|
||||||
### Step 5: Check Deployment Status
|
### Step 6: Check Deployment Status
|
||||||
|
|
||||||
After the deploy completes, verify with the built-in status command:
|
After the deploy completes, verify with the built-in status command:
|
||||||
|
|
||||||
|
|
@ -212,7 +222,7 @@ After the deploy completes, verify with the built-in status command:
|
||||||
- Port 8443 is open
|
- Port 8443 is open
|
||||||
- Incus remote status shown (if configured)
|
- Incus remote status shown (if configured)
|
||||||
|
|
||||||
### Step 6: Verify the Deployed VM
|
### Step 7: Verify the Deployed VM
|
||||||
|
|
||||||
Once the script reports success with an IP address:
|
Once the script reports success with an IP address:
|
||||||
|
|
||||||
|
|
@ -233,7 +243,7 @@ incus network list incus-test:
|
||||||
ARP-based lookup (MAC from Proxmox VM config → ARP table). If the IP is wrong,
|
ARP-based lookup (MAC from Proxmox VM config → ARP table). If the IP is wrong,
|
||||||
check the Proxmox console — stale ARP entries can be misleading.
|
check the Proxmox console — stale ARP entries can be misleading.
|
||||||
|
|
||||||
### Step 7: Test Re-run Reconcile
|
### Step 8: Test Re-run Reconcile
|
||||||
|
|
||||||
Re-run the deploy to verify the reconcile menu:
|
Re-run the deploy to verify the reconcile menu:
|
||||||
|
|
||||||
|
|
@ -252,7 +262,7 @@ Also test with `--yes` to verify it defaults to safe behavior:
|
||||||
|
|
||||||
**Verify:** Runs post-deployment checks, does NOT destroy.
|
**Verify:** Runs post-deployment checks, does NOT destroy.
|
||||||
|
|
||||||
### Step 8: Clean Up the Test VM
|
### Step 9: Clean Up the Test VM
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
./incusos/incusos-proxmox --cleanup my-test.yaml
|
./incusos/incusos-proxmox --cleanup my-test.yaml
|
||||||
|
|
@ -269,7 +279,7 @@ Remove the remote too:
|
||||||
incus remote remove incus-test
|
incus remote remove incus-test
|
||||||
```
|
```
|
||||||
|
|
||||||
### Step 9: (Optional) Test API Method
|
### Step 10: (Optional) Test API Method
|
||||||
|
|
||||||
After single-VM success with SSH, switch to API tokens.
|
After single-VM success with SSH, switch to API tokens.
|
||||||
|
|
||||||
|
|
@ -302,9 +312,9 @@ proxmox:
|
||||||
api_token_id: automation@pve!deploy
|
api_token_id: automation@pve!deploy
|
||||||
```
|
```
|
||||||
|
|
||||||
Repeat steps 3-6 using the API method.
|
Repeat steps 4-7 using the API method.
|
||||||
|
|
||||||
### Step 10: (Optional) Test with Resource Pool
|
### Step 11: (Optional) Test with Resource Pool
|
||||||
|
|
||||||
For API method with pool isolation:
|
For API method with pool isolation:
|
||||||
|
|
||||||
|
|
@ -331,7 +341,7 @@ Deploy and verify:
|
||||||
./incusos/incusos-proxmox --status my-test.yaml # Status scoped to pool
|
./incusos/incusos-proxmox --status my-test.yaml # Status scoped to pool
|
||||||
```
|
```
|
||||||
|
|
||||||
### Step 11: (Optional) Full Lab Deployment
|
### Step 12: (Optional) Full Lab Deployment
|
||||||
|
|
||||||
Once single-VM tests pass, deploy the full 4-VM lab:
|
Once single-VM tests pass, deploy the full 4-VM lab:
|
||||||
|
|
||||||
|
|
@ -392,32 +402,229 @@ incus remote add incus-lab-03 <IP3> --accept-certificate
|
||||||
|
|
||||||
### Manual Cluster Formation (Alternative)
|
### Manual Cluster Formation (Alternative)
|
||||||
|
|
||||||
If you prefer to form the cluster manually instead of using `lab-test`:
|
If you prefer to form the cluster manually instead of using `lab-test`,
|
||||||
|
follow the steps below. See also `notes/clustering-guide.md` for the full
|
||||||
|
reference guide with explanations of what happens under the hood.
|
||||||
|
|
||||||
|
#### Prerequisites
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Step 1: Enable clustering on first node
|
# Check client version (need 6.20+ for remote cluster join)
|
||||||
incus cluster enable incus-lab-01: incus-lab-01
|
incus version
|
||||||
|
|
||||||
# Step 2: Generate join token for second node
|
# Verify all three remotes are working
|
||||||
incus cluster add incus-lab-01: incus-lab-02
|
incus info incus-lab-01: | head -5
|
||||||
|
incus info incus-lab-02: | head -5
|
||||||
|
incus info incus-lab-03: | head -5
|
||||||
|
```
|
||||||
|
|
||||||
# Step 3: Join second node
|
#### Step 1: Set specific IPs on all nodes
|
||||||
incus cluster join incus-lab-01: incus-lab-02:
|
|
||||||
|
|
||||||
# Step 4: Repeat for third node
|
IncusOS defaults to `core.https_address: :8443` (wildcard). Clustering
|
||||||
incus cluster add incus-lab-01: incus-lab-03
|
requires a specific routable IP on each node.
|
||||||
incus cluster join incus-lab-01: incus-lab-03:
|
|
||||||
|
|
||||||
# Step 5: Verify
|
```bash
|
||||||
|
# Discover each node's routable IP
|
||||||
|
for node in incus-lab-01 incus-lab-02 incus-lab-03; do
|
||||||
|
echo "=== $node ==="
|
||||||
|
incus query "$node:/1.0" | python3 -c "import sys,json; d=json.load(sys.stdin); \
|
||||||
|
[print(a) for a in d['environment']['addresses'] \
|
||||||
|
if not a.startswith('10.') and not a.startswith('fd42:') and not a.startswith('[')]"
|
||||||
|
done
|
||||||
|
|
||||||
|
# Set the specific IP on each node (replace with actual IPs)
|
||||||
|
incus config set incus-lab-01: core.https_address <IP1>:8443
|
||||||
|
incus config set incus-lab-02: core.https_address <IP2>:8443
|
||||||
|
incus config set incus-lab-03: core.https_address <IP3>:8443
|
||||||
|
|
||||||
|
# Verify connectivity still works after each change
|
||||||
|
incus info incus-lab-01: | head -3
|
||||||
|
incus info incus-lab-02: | head -3
|
||||||
|
incus info incus-lab-03: | head -3
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Step 2: Enable clustering on the init node
|
||||||
|
|
||||||
|
```bash
|
||||||
|
incus cluster enable incus-lab-01:incus-lab-01
|
||||||
|
```
|
||||||
|
|
||||||
|
> **Important**: This regenerates the server's TLS certificate. Your remote
|
||||||
|
> will break with a certificate error. Fix it:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
incus remote switch local # if incus-lab-01 is the current default
|
||||||
|
incus remote remove incus-lab-01
|
||||||
|
incus remote add incus-lab-01 https://<IP1>:8443 --accept-certificate
|
||||||
|
```
|
||||||
|
|
||||||
|
Verify:
|
||||||
|
|
||||||
|
```bash
|
||||||
incus cluster list incus-lab-01:
|
incus cluster list incus-lab-01:
|
||||||
|
# Should show one member (incus-lab-01), status ONLINE, role database-leader
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Step 3: Prepare joining nodes
|
||||||
|
|
||||||
|
Nodes with `apply_defaults: true` have a pre-existing `local` storage pool
|
||||||
|
and `incusbr0` network. These conflict with the cluster join process (the
|
||||||
|
join wizard tries to set member-specific ZFS config keys at cluster level).
|
||||||
|
Delete them before joining:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# For each joining node (incus-lab-02, incus-lab-03):
|
||||||
|
NODE=incus-lab-02 # then repeat with incus-lab-03
|
||||||
|
|
||||||
|
# Remove server-level references to the pool
|
||||||
|
incus config unset "$NODE": storage.backups_volume
|
||||||
|
incus config unset "$NODE": storage.images_volume
|
||||||
|
|
||||||
|
# Delete the backup and image volumes
|
||||||
|
incus storage volume delete "$NODE":local backups
|
||||||
|
incus storage volume delete "$NODE":local images
|
||||||
|
|
||||||
|
# Clear default profile references (otherwise pool is "in use")
|
||||||
|
incus profile device remove "$NODE":default root
|
||||||
|
incus profile device remove "$NODE":default eth0
|
||||||
|
|
||||||
|
# Delete pool and network
|
||||||
|
incus storage delete "$NODE":local
|
||||||
|
incus network delete "$NODE":incusbr0
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Step 4: Join nodes to the cluster
|
||||||
|
|
||||||
|
**Interactive** (one node at a time):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Generate join token
|
||||||
|
incus cluster add incus-lab-01:incus-lab-02
|
||||||
|
|
||||||
|
# Join (answer the interactive prompts)
|
||||||
|
incus cluster join incus-lab-01: incus-lab-02:
|
||||||
|
# Prompt 1 - IP address: press Enter (accept default)
|
||||||
|
# Prompt 2 - Member name: press Enter (accept default)
|
||||||
|
# Prompt 3 - "All existing data is lost": type "yes"
|
||||||
|
# Prompt 4 - source for storage pool "local": type "local/incus"
|
||||||
|
# Prompt 5 - zfs.pool_name for storage pool "local": type "local/incus"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Automated** (non-interactive):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Generate token + join in one go
|
||||||
|
incus cluster add incus-lab-01:incus-lab-02
|
||||||
|
printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join incus-lab-01: incus-lab-02:
|
||||||
|
```
|
||||||
|
|
||||||
|
After each join, fix the remote (same certificate issue as the init node):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
incus remote remove incus-lab-02
|
||||||
|
incus remote add incus-lab-02 https://<IP2>:8443 --accept-certificate
|
||||||
|
```
|
||||||
|
|
||||||
|
Repeat for incus-lab-03.
|
||||||
|
|
||||||
|
#### Step 5: Verify the cluster
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# All 3 members should be ONLINE
|
||||||
|
incus cluster list incus-lab-01:
|
||||||
|
|
||||||
|
# Storage pool should show on all members
|
||||||
|
incus storage show incus-lab-01:local
|
||||||
|
incus storage show incus-lab-01:local --target incus-lab-01
|
||||||
|
incus storage show incus-lab-01:local --target incus-lab-02
|
||||||
|
incus storage show incus-lab-01:local --target incus-lab-03
|
||||||
|
|
||||||
|
# All remotes should work
|
||||||
|
incus info incus-lab-01: | head -3
|
||||||
|
incus info incus-lab-02: | head -3
|
||||||
|
incus info incus-lab-03: | head -3
|
||||||
```
|
```
|
||||||
|
|
||||||
**Key points:**
|
**Key points:**
|
||||||
- `apply_defaults: true` on all nodes ensures matching storage pool and
|
- `apply_defaults: true` on all nodes ensures matching storage pool and
|
||||||
network bridge configurations (required for migration)
|
network bridge names (required for migration) -- but the pools must be
|
||||||
|
deleted on joining nodes before the join
|
||||||
- After joining, manage the cluster through the init node's remote
|
- After joining, manage the cluster through the init node's remote
|
||||||
|
- The ZFS pool source is `local/incus` on IncusOS (auto-created by
|
||||||
|
`apply_defaults`)
|
||||||
|
- Each join generates a new token (tokens are single-use and time-limited)
|
||||||
- VM migration requires matching storage pool names across nodes
|
- VM migration requires matching storage pool names across nodes
|
||||||
|
|
||||||
|
### Migration Testing (After Cluster Formation)
|
||||||
|
|
||||||
|
Once the cluster is formed, validate container and VM migration:
|
||||||
|
|
||||||
|
#### Container migration (stop/move/start)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Launch on node 2
|
||||||
|
incus launch images:debian/12 incus-lab-01:test-container --target incus-lab-02
|
||||||
|
incus exec incus-lab-01:test-container -- bash -c 'echo "hello" > /root/test.txt'
|
||||||
|
|
||||||
|
# Migrate to node 3
|
||||||
|
incus stop incus-lab-01:test-container
|
||||||
|
incus move incus-lab-01:test-container --target incus-lab-03
|
||||||
|
incus start incus-lab-01:test-container
|
||||||
|
|
||||||
|
# Verify: data survives, processes don't
|
||||||
|
incus exec incus-lab-01:test-container -- cat /root/test.txt # "hello"
|
||||||
|
incus list incus-lab-01: --columns nLs # node 3
|
||||||
|
```
|
||||||
|
|
||||||
|
#### VM live migration
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Launch and configure for migration
|
||||||
|
incus launch images:debian/12 incus-lab-01:test-vm --vm --target incus-lab-02
|
||||||
|
# Wait for VM agent...
|
||||||
|
incus stop incus-lab-01:test-vm
|
||||||
|
incus config set incus-lab-01:test-vm migration.stateful=true
|
||||||
|
incus config set incus-lab-01:test-vm limits.cpu=0-1
|
||||||
|
incus config device add incus-lab-01:test-vm root disk path=/ pool=local size.state=2GiB
|
||||||
|
incus start incus-lab-01:test-vm
|
||||||
|
|
||||||
|
# Set up heartbeat for state verification
|
||||||
|
incus exec incus-lab-01:test-vm -- mkdir -p /tmp/migration-test
|
||||||
|
incus exec incus-lab-01:test-vm -- bash -c \
|
||||||
|
'cat > /tmp/migration-test/heartbeat.sh << "EOF"
|
||||||
|
#!/bin/bash
|
||||||
|
i=0; while true; do echo $i > /tmp/migration-test/heartbeat; i=$((i+1)); sleep 1; done
|
||||||
|
EOF
|
||||||
|
chmod +x /tmp/migration-test/heartbeat.sh'
|
||||||
|
incus exec incus-lab-01:test-vm -- \
|
||||||
|
systemd-run --unit=migration-heartbeat /tmp/migration-test/heartbeat.sh
|
||||||
|
|
||||||
|
# Live-migrate (state preserved)
|
||||||
|
sleep 3
|
||||||
|
incus exec incus-lab-01:test-vm -- cat /tmp/migration-test/heartbeat # note value
|
||||||
|
incus move incus-lab-01:test-vm --target incus-lab-03
|
||||||
|
sleep 2 # agent needs ~1-2s to reconnect
|
||||||
|
incus exec incus-lab-01:test-vm -- cat /tmp/migration-test/heartbeat # should be higher
|
||||||
|
incus exec incus-lab-01:test-vm -- systemctl is-active migration-heartbeat # "active"
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Cluster evacuation
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Move both workloads to node 2, then evacuate
|
||||||
|
incus cluster evacuate incus-lab-01:incus-lab-02 --force
|
||||||
|
incus cluster list incus-lab-01: # node 2 = EVACUATED
|
||||||
|
incus list incus-lab-01: --columns nLs # workloads on other nodes
|
||||||
|
|
||||||
|
# Restore
|
||||||
|
incus cluster restore incus-lab-01:incus-lab-02 --force
|
||||||
|
incus cluster list incus-lab-01: # node 2 = ONLINE
|
||||||
|
incus list incus-lab-01: --columns nLs # workloads back on node 2
|
||||||
|
```
|
||||||
|
|
||||||
|
See `notes/clustering-guide.md` for the full automated migration test script
|
||||||
|
and detailed test results.
|
||||||
|
|
||||||
### Cleanup
|
### Cleanup
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
|
@ -427,8 +634,12 @@ incus cluster list incus-lab-01:
|
||||||
# Destroy VMs only
|
# Destroy VMs only
|
||||||
./incusos/incusos-proxmox --cleanup examples/lab-cluster.yaml
|
./incusos/incusos-proxmox --cleanup examples/lab-cluster.yaml
|
||||||
|
|
||||||
# Full cleanup: VMs + ISOs + seeds + remotes + cache
|
# Full cleanup: VMs + deployment ISO + seeds + remotes + cache
|
||||||
./incusos/incusos-proxmox --cleanup --deep --yes examples/lab-cluster.yaml
|
./incusos/incusos-proxmox --cleanup --deep --yes examples/lab-cluster.yaml
|
||||||
|
|
||||||
|
# Pool-wide: destroy ALL managed VMs (no config file needed)
|
||||||
|
./incusos/incusos-proxmox --cleanup-all --yes
|
||||||
|
./incusos/incusos-proxmox --cleanup-all --deep --yes # nuke everything
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
@ -507,3 +718,7 @@ CLI is installed.
|
||||||
| Pool not found | Resource pool not created | Run `pveum pool add <name>` on Proxmox |
|
| Pool not found | Resource pool not created | Run `pveum pool add <name>` on Proxmox |
|
||||||
| Re-run destroys VMs | Used option 3 in reconcile | Use option 1 (checks) or option 2 (continue) instead |
|
| Re-run destroys VMs | Used option 3 in reconcile | Use option 1 (checks) or option 2 (continue) instead |
|
||||||
| `--status` shows "not found" | VM was cleaned up | Deploy first, then check status |
|
| `--status` shows "not found" | VM was cleaned up | Deploy first, then check status |
|
||||||
|
| VM live migration fails: "Missing section footer for ICH9LPC" | `limits.cpu` not set or set as integer | Use range syntax: `limits.cpu=0-1` (not `2`); see `notes/clustering-guide.md` |
|
||||||
|
| `migration.stateful` cannot be set while running | Incus requires VM to be stopped | Stop VM first, set config, then start |
|
||||||
|
| `certificate is valid for 127.0.0.1, ::1, not <IP>` | Cluster cert regenerated on enable/join | Remove and re-add remote with `--accept-certificate` |
|
||||||
|
| `Config key "zfs.pool_name" is cluster member specific` | Storage pool exists on joining node | Delete pool/network before joining; see clustering guide |
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,45 @@
|
||||||
|
# lab-advanced.yaml - 3-node heterogeneous cluster for advanced migration testing
|
||||||
|
#
|
||||||
|
# Deploys 3 Incus nodes with a 6/4/4 core split to test VM live migration
|
||||||
|
# across hosts with different CPU topologies. 8 GiB RAM provides headroom
|
||||||
|
# for multiple concurrent VMs with size.state for stateful operations.
|
||||||
|
#
|
||||||
|
# Test matrix targets:
|
||||||
|
# - Multi-vCPU VMs (2, 3, 4 vCPUs) with limits.cpu ranges
|
||||||
|
# - Concurrent migrations from different source nodes
|
||||||
|
# - Edge cases: active I/O, memory-loaded VMs, 4-vCPU on 4-core host
|
||||||
|
# - Cluster lifecycle: evacuate, remove, replace, rejoin, rebalance
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# incusos-proxmox --dry-run examples/lab-advanced.yaml # Preview
|
||||||
|
# incusos-proxmox --yes examples/lab-advanced.yaml # Deploy
|
||||||
|
# incusos-proxmox --status examples/lab-advanced.yaml # Check status
|
||||||
|
# incusos-proxmox --cleanup examples/lab-advanced.yaml # Tear down VMs
|
||||||
|
# incusos-proxmox --cleanup --deep examples/lab-advanced.yaml # Full cleanup
|
||||||
|
#
|
||||||
|
# Connection settings from proxmox.yaml (see examples/proxmox.yaml.example).
|
||||||
|
# Add a proxmox: section here to override per-deployment.
|
||||||
|
#
|
||||||
|
# Note: start_vmid 910 may collide with lab-oc.yaml. Clean up old deployments
|
||||||
|
# first or adjust the range.
|
||||||
|
|
||||||
|
defaults:
|
||||||
|
memory: 8192
|
||||||
|
disk: 50
|
||||||
|
start_vmid: 910
|
||||||
|
|
||||||
|
vms:
|
||||||
|
- name: incus-adv-01
|
||||||
|
app: incus
|
||||||
|
apply_defaults: true
|
||||||
|
cores: 6
|
||||||
|
|
||||||
|
- name: incus-adv-02
|
||||||
|
app: incus
|
||||||
|
apply_defaults: true
|
||||||
|
cores: 4
|
||||||
|
|
||||||
|
- name: incus-adv-03
|
||||||
|
app: incus
|
||||||
|
apply_defaults: true
|
||||||
|
cores: 4
|
||||||
|
|
@ -13,16 +13,12 @@
|
||||||
#
|
#
|
||||||
# After deployment, form a cluster with lab-test:
|
# After deployment, form a cluster with lab-test:
|
||||||
# lab-test --phase cluster examples/lab-cluster.yaml
|
# lab-test --phase cluster examples/lab-cluster.yaml
|
||||||
|
#
|
||||||
proxmox:
|
# Connection settings from proxmox.yaml (see examples/proxmox.yaml.example).
|
||||||
host: 192.168.1.29
|
# Add a proxmox: section here to override per-deployment.
|
||||||
method: api
|
|
||||||
api_token_id: automation@pve!deploy
|
|
||||||
storage: local-zfs
|
|
||||||
pool: IncusLab
|
|
||||||
|
|
||||||
defaults:
|
defaults:
|
||||||
cores: 2
|
cores: 4
|
||||||
memory: 4096
|
memory: 4096
|
||||||
disk: 50
|
disk: 50
|
||||||
start_vmid: 900
|
start_vmid: 900
|
||||||
|
|
@ -31,9 +27,6 @@ vms:
|
||||||
- name: incus-lab-01
|
- name: incus-lab-01
|
||||||
app: incus
|
app: incus
|
||||||
apply_defaults: true
|
apply_defaults: true
|
||||||
cores: 4
|
|
||||||
memory: 8192
|
|
||||||
disk: 64
|
|
||||||
|
|
||||||
- name: incus-lab-02
|
- name: incus-lab-02
|
||||||
app: incus
|
app: incus
|
||||||
|
|
|
||||||
|
|
@ -15,13 +15,9 @@
|
||||||
# incusos-proxmox --yes examples/lab-oc.yaml
|
# incusos-proxmox --yes examples/lab-oc.yaml
|
||||||
# incusos-proxmox --status examples/lab-oc.yaml
|
# incusos-proxmox --status examples/lab-oc.yaml
|
||||||
# incusos-proxmox --cleanup --deep examples/lab-oc.yaml
|
# incusos-proxmox --cleanup --deep examples/lab-oc.yaml
|
||||||
|
#
|
||||||
proxmox:
|
# Connection settings from proxmox.yaml (see examples/proxmox.yaml.example).
|
||||||
host: 192.168.1.29
|
# Add a proxmox: section here to override per-deployment.
|
||||||
method: api
|
|
||||||
api_token_id: automation@pve!deploy
|
|
||||||
storage: local-zfs
|
|
||||||
pool: IncusLab
|
|
||||||
|
|
||||||
defaults:
|
defaults:
|
||||||
cores: 2
|
cores: 2
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,24 @@
|
||||||
|
# lab-replace.yaml - Single replacement node for cluster lifecycle testing
|
||||||
|
#
|
||||||
|
# Used in Part 4 of the advanced migration test plan: after evacuating,
|
||||||
|
# removing, and destroying incus-adv-03, this config deploys a fresh
|
||||||
|
# replacement node that gets joined back into the cluster.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# incusos-proxmox --dry-run examples/lab-replace.yaml
|
||||||
|
# incusos-proxmox --yes examples/lab-replace.yaml
|
||||||
|
# incusos-proxmox --status examples/lab-replace.yaml
|
||||||
|
# incusos-proxmox --cleanup examples/lab-replace.yaml
|
||||||
|
#
|
||||||
|
# Connection settings from proxmox.yaml (see examples/proxmox.yaml.example).
|
||||||
|
|
||||||
|
defaults:
|
||||||
|
memory: 8192
|
||||||
|
disk: 50
|
||||||
|
start_vmid: 913
|
||||||
|
|
||||||
|
vms:
|
||||||
|
- name: incus-adv-04
|
||||||
|
app: incus
|
||||||
|
apply_defaults: true
|
||||||
|
cores: 4
|
||||||
|
|
@ -1,9 +1,5 @@
|
||||||
proxmox:
|
# Connection settings from proxmox.yaml (see examples/proxmox.yaml.example).
|
||||||
host: 192.168.1.29
|
# Add a proxmox: section here to override per-deployment.
|
||||||
method: api
|
|
||||||
api_token_id: automation@pve!deploy
|
|
||||||
storage: local-zfs # VM disk storage (default: local-lvm)
|
|
||||||
pool: IncusLab
|
|
||||||
|
|
||||||
defaults:
|
defaults:
|
||||||
cores: 2
|
cores: 2
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,21 @@
|
||||||
|
# proxmox.yaml.example - Proxmox connection config template
|
||||||
|
#
|
||||||
|
# Copy to incusos/proxmox.yaml and edit with your settings.
|
||||||
|
# This file is auto-discovered by incusos-proxmox and lab-test.
|
||||||
|
#
|
||||||
|
# Lab config files (lab-cluster.yaml, lab-oc.yaml, etc.) reference
|
||||||
|
# this for connection settings, so you only need to change credentials
|
||||||
|
# in one place.
|
||||||
|
#
|
||||||
|
# Override any setting per-deployment by adding a proxmox: section
|
||||||
|
# to the lab config file.
|
||||||
|
|
||||||
|
host: 192.168.1.10 # Proxmox host IP or hostname
|
||||||
|
method: ssh # ssh (default) or api
|
||||||
|
# api_token_id: automation@pve!deploy # For api method
|
||||||
|
node: pve # Proxmox node name
|
||||||
|
storage: local-lvm # VM disk storage
|
||||||
|
iso_storage: local # ISO/image storage
|
||||||
|
bridge: vmbr0 # Network bridge
|
||||||
|
# pool: IncusLab # Resource pool (optional, for isolation)
|
||||||
|
ssh_user: root # SSH user (for ssh method)
|
||||||
|
|
@ -43,7 +43,7 @@ success() { [[ "$QUIET" == true ]] && return; echo -e "${GREEN}[ok]${RESET}
|
||||||
warn() { echo -e "${YELLOW}[warn]${RESET} $*" >&2; }
|
warn() { echo -e "${YELLOW}[warn]${RESET} $*" >&2; }
|
||||||
error() { echo -e "${RED}[error]${RESET} $*" >&2; }
|
error() { echo -e "${RED}[error]${RESET} $*" >&2; }
|
||||||
step() { [[ "$QUIET" == true ]] && return; echo -e "${CYAN}[step]${RESET} ${BOLD}$*${RESET}"; }
|
step() { [[ "$QUIET" == true ]] && return; echo -e "${CYAN}[step]${RESET} ${BOLD}$*${RESET}"; }
|
||||||
detail() { [[ "$QUIET" == true ]] && return; echo -e "${DIM} $*${RESET}"; }
|
detail() { [[ "$VERBOSE" != true ]] && return; echo -e "${DIM} $*${RESET}"; }
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Help
|
# Help
|
||||||
|
|
@ -91,25 +91,39 @@ ${BOLD}OPTIONS${RESET}
|
||||||
${BOLD}--cleanup${RESET} Destroy VMs defined in config (managed only)
|
${BOLD}--cleanup${RESET} Destroy VMs defined in config (managed only)
|
||||||
${BOLD}--cleanup --deep${RESET} Also remove ISOs, seeds, remotes, and local cache
|
${BOLD}--cleanup --deep${RESET} Also remove ISOs, seeds, remotes, and local cache
|
||||||
${BOLD}--cleanup-all${RESET} Destroy ALL managed VMs in the pool
|
${BOLD}--cleanup-all${RESET} Destroy ALL managed VMs in the pool
|
||||||
|
${BOLD}--proxmox${RESET} FILE Proxmox connection config (default: auto-discover)
|
||||||
${BOLD}--force-cleanup${RESET} Destroy VMs even without management marker
|
${BOLD}--force-cleanup${RESET} Destroy VMs even without management marker
|
||||||
${BOLD}--dry-run${RESET} Preview actions without executing
|
${BOLD}--dry-run${RESET} Preview actions without executing
|
||||||
|
${BOLD}-v, --verbose${RESET} Show detailed output (tool paths, API calls)
|
||||||
${BOLD}-q, --quiet${RESET} Suppress informational output
|
${BOLD}-q, --quiet${RESET} Suppress informational output
|
||||||
${BOLD}-h, --help${RESET} Show this help message
|
${BOLD}-h, --help${RESET} Show this help message
|
||||||
${BOLD}-V, --version${RESET} Show version
|
${BOLD}-V, --version${RESET} Show version
|
||||||
|
|
||||||
|
${BOLD}PROXMOX CONNECTION CONFIG${RESET}
|
||||||
|
Proxmox connection settings can be stored in a separate proxmox.yaml file
|
||||||
|
to avoid duplicating credentials across lab configs. Auto-discovered in:
|
||||||
|
script directory, then current directory. Override with --proxmox FILE.
|
||||||
|
|
||||||
|
${BOLD}proxmox.yaml${RESET} (flat structure, no wrapping key):
|
||||||
|
host: 192.168.1.10 # Proxmox host (required)
|
||||||
|
method: api # ssh or api (default: ssh)
|
||||||
|
api_token_id: user!token # API token (for api method)
|
||||||
|
node: pve # Node name (default: pve)
|
||||||
|
storage: local-lvm # VM disk storage (default: local-lvm)
|
||||||
|
iso_storage: local # ISO storage (default: local)
|
||||||
|
bridge: vmbr0 # Network bridge (default: vmbr0)
|
||||||
|
pool: IncusLab # Resource pool (optional)
|
||||||
|
ssh_user: root # SSH user (default: root)
|
||||||
|
|
||||||
|
Merge priority: proxmox.yaml → lab config proxmox: section → CLI flags
|
||||||
|
|
||||||
${BOLD}CONFIG FILE${RESET}
|
${BOLD}CONFIG FILE${RESET}
|
||||||
YAML file defining the Proxmox connection and VM specifications.
|
YAML file defining VM specifications (and optional Proxmox overrides).
|
||||||
See examples/proxmox-lab.yaml for a full example.
|
See examples/proxmox-lab.yaml for a full example.
|
||||||
|
|
||||||
${BOLD}proxmox:${RESET}
|
${BOLD}proxmox:${RESET} # Optional: overrides proxmox.yaml
|
||||||
host: 192.168.1.10 # Proxmox host (required)
|
host: 192.168.1.10 # Proxmox host
|
||||||
node: pve # Node name (default: pve)
|
storage: local-zfs # Override storage for this deployment
|
||||||
storage: local-lvm # VM disk storage (default: local-lvm)
|
|
||||||
iso_storage: local # ISO storage (default: local)
|
|
||||||
bridge: vmbr0 # Network bridge (default: vmbr0)
|
|
||||||
pool: IncusLab # Resource pool (optional)
|
|
||||||
method: ssh # ssh or api (default: ssh)
|
|
||||||
ssh_user: root # SSH user (default: root)
|
|
||||||
|
|
||||||
${BOLD}defaults:${RESET}
|
${BOLD}defaults:${RESET}
|
||||||
cores: 4 # Default CPU cores
|
cores: 4 # Default CPU cores
|
||||||
|
|
@ -152,16 +166,20 @@ ${BOLD}EXAMPLES${RESET}
|
||||||
# Destroy all VMs defined in config
|
# Destroy all VMs defined in config
|
||||||
${SCRIPT_NAME} --cleanup examples/proxmox-lab.yaml
|
${SCRIPT_NAME} --cleanup examples/proxmox-lab.yaml
|
||||||
|
|
||||||
# Deep cleanup: VMs + ISOs + seeds + remotes + cache
|
# Deep cleanup: VMs + the IncusOS ISO used + seeds + remotes + cache
|
||||||
${SCRIPT_NAME} --cleanup --deep examples/proxmox-lab.yaml
|
${SCRIPT_NAME} --cleanup --deep examples/proxmox-lab.yaml
|
||||||
|
|
||||||
# Destroy ALL managed VMs in the pool (not config-specific)
|
# Destroy ALL managed VMs in the pool (uses proxmox.yaml, no config needed)
|
||||||
${SCRIPT_NAME} --cleanup-all examples/proxmox-lab.yaml
|
${SCRIPT_NAME} --cleanup-all
|
||||||
|
${SCRIPT_NAME} --cleanup-all --deep # also remove all ISOs + seeds
|
||||||
|
|
||||||
# Check tool versions and environment health
|
# Check tool versions and environment health
|
||||||
${SCRIPT_NAME} --doctor
|
${SCRIPT_NAME} --doctor
|
||||||
${SCRIPT_NAME} --doctor examples/proxmox-lab.yaml # also check Proxmox
|
${SCRIPT_NAME} --doctor examples/proxmox-lab.yaml # also check Proxmox
|
||||||
|
|
||||||
|
# Verbose output (show dependency paths, API calls, per-file actions)
|
||||||
|
${SCRIPT_NAME} --doctor -v
|
||||||
|
|
||||||
# Use API method with token
|
# Use API method with token
|
||||||
export PROXMOX_TOKEN_SECRET="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
|
export PROXMOX_TOKEN_SECRET="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
|
||||||
${SCRIPT_NAME} --method api examples/proxmox-lab.yaml
|
${SCRIPT_NAME} --method api examples/proxmox-lab.yaml
|
||||||
|
|
@ -228,6 +246,8 @@ DEEP_CLEANUP=false
|
||||||
FORCE_CLEANUP=false
|
FORCE_CLEANUP=false
|
||||||
DRY_RUN=false
|
DRY_RUN=false
|
||||||
QUIET=false
|
QUIET=false
|
||||||
|
VERBOSE=false
|
||||||
|
PROXMOX_CONFIG=""
|
||||||
CONFIG_FILE=""
|
CONFIG_FILE=""
|
||||||
|
|
||||||
parse_args() {
|
parse_args() {
|
||||||
|
|
@ -290,10 +310,18 @@ parse_args() {
|
||||||
FORCE_CLEANUP=true
|
FORCE_CLEANUP=true
|
||||||
shift
|
shift
|
||||||
;;
|
;;
|
||||||
|
--proxmox)
|
||||||
|
PROXMOX_CONFIG="${2:?'--proxmox requires a file path'}"
|
||||||
|
shift 2
|
||||||
|
;;
|
||||||
--dry-run)
|
--dry-run)
|
||||||
DRY_RUN=true
|
DRY_RUN=true
|
||||||
shift
|
shift
|
||||||
;;
|
;;
|
||||||
|
-v|--verbose)
|
||||||
|
VERBOSE=true
|
||||||
|
shift
|
||||||
|
;;
|
||||||
-q|--quiet)
|
-q|--quiet)
|
||||||
QUIET=true
|
QUIET=true
|
||||||
shift
|
shift
|
||||||
|
|
@ -329,11 +357,22 @@ parse_args() {
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
validate_args() {
|
validate_args() {
|
||||||
|
# Universal checks (apply regardless of mode)
|
||||||
|
if [[ "$VERBOSE" == true ]] && [[ "$QUIET" == true ]]; then
|
||||||
|
error "Cannot use both --verbose and --quiet"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
# --doctor can run without a config file
|
# --doctor can run without a config file
|
||||||
if [[ "$DOCTOR" == true ]] && [[ -z "$CONFIG_FILE" ]]; then
|
if [[ "$DOCTOR" == true ]] && [[ -z "$CONFIG_FILE" ]]; then
|
||||||
return 0
|
return 0
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
# --cleanup-all can run without a config file (uses proxmox.yaml)
|
||||||
|
if [[ "$CLEANUP_ALL" == true ]] && [[ -z "$CONFIG_FILE" ]]; then
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
|
||||||
if [[ -z "$CONFIG_FILE" ]]; then
|
if [[ -z "$CONFIG_FILE" ]]; then
|
||||||
error "No configuration file specified"
|
error "No configuration file specified"
|
||||||
echo "Usage: ${SCRIPT_NAME} [OPTIONS] CONFIG_FILE" >&2
|
echo "Usage: ${SCRIPT_NAME} [OPTIONS] CONFIG_FILE" >&2
|
||||||
|
|
@ -605,7 +644,19 @@ if versions:
|
||||||
echo -e " Cached ISO: ${DIM}none${RESET}"
|
echo -e " Cached ISO: ${DIM}none${RESET}"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
# -- Proxmox section (only if config file provided) --
|
# -- Config section --
|
||||||
|
echo ""
|
||||||
|
echo -e " ${BOLD}Config:${RESET}"
|
||||||
|
if find_proxmox_config; then
|
||||||
|
echo -e " proxmox.yaml: ${GREEN}${PROXMOX_CONFIG_RESOLVED} ✓${RESET}"
|
||||||
|
else
|
||||||
|
echo -e " proxmox.yaml: ${DIM}not found (optional)${RESET}"
|
||||||
|
fi
|
||||||
|
if [[ -n "$CONFIG_FILE" ]] && [[ -f "$CONFIG_FILE" ]]; then
|
||||||
|
echo -e " Lab config: ${CONFIG_FILE}"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# -- Proxmox section (if config file or proxmox.yaml provides connection info) --
|
||||||
echo ""
|
echo ""
|
||||||
if [[ -n "$CONFIG_FILE" ]] && [[ -f "$CONFIG_FILE" ]]; then
|
if [[ -n "$CONFIG_FILE" ]] && [[ -f "$CONFIG_FILE" ]]; then
|
||||||
echo -e " ${BOLD}Proxmox:${RESET}"
|
echo -e " ${BOLD}Proxmox:${RESET}"
|
||||||
|
|
@ -713,9 +764,68 @@ json.dump(data, sys.stdout, indent=2)
|
||||||
echo -e " ${RED}Failed to parse config file${RESET}"
|
echo -e " ${RED}Failed to parse config file${RESET}"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
rm -rf "$doctor_workdir"
|
||||||
|
elif [[ -n "$PROXMOX_CONFIG_RESOLVED" ]]; then
|
||||||
|
# Use proxmox.yaml for connectivity check (no lab config provided)
|
||||||
|
echo -e " ${BOLD}Proxmox:${RESET} ${DIM}(from proxmox.yaml)${RESET}"
|
||||||
|
|
||||||
|
local doctor_workdir
|
||||||
|
doctor_workdir=$(mktemp -d "${TMPDIR:-/tmp}/${SCRIPT_NAME}-doctor-XXXXXX")
|
||||||
|
WORKDIR="$doctor_workdir"
|
||||||
|
|
||||||
|
load_proxmox_config "$PROXMOX_CONFIG_RESOLVED"
|
||||||
|
[[ -z "$PVE_NODE" ]] && PVE_NODE="pve"
|
||||||
|
[[ -z "$PVE_METHOD" ]] && PVE_METHOD="ssh"
|
||||||
|
[[ -z "$PVE_SSH_USER" ]] && PVE_SSH_USER="root"
|
||||||
|
[[ -n "$METHOD" ]] && PVE_METHOD="$METHOD"
|
||||||
|
[[ -n "$HOST_OVERRIDE" ]] && PVE_HOST="$HOST_OVERRIDE"
|
||||||
|
|
||||||
|
if [[ -z "$PVE_HOST" ]]; then
|
||||||
|
echo -e " Host: ${RED}not set in proxmox.yaml${RESET}"
|
||||||
|
else
|
||||||
|
echo -e " Host: ${PVE_HOST} (${PVE_METHOD})"
|
||||||
|
local pve_ok=false
|
||||||
|
if [[ "$PVE_METHOD" == "api" ]]; then
|
||||||
|
if [[ -n "$PVE_API_TOKEN_ID" ]] && [[ -n "${PROXMOX_TOKEN_SECRET:-}" ]]; then
|
||||||
|
local auth="Authorization: PVEAPIToken=${PVE_API_TOKEN_ID}=${PROXMOX_TOKEN_SECRET}"
|
||||||
|
local pve_ver
|
||||||
|
pve_ver=$(curl -fsSk --connect-timeout 5 -H "$auth" \
|
||||||
|
"https://${PVE_HOST}:8006/api2/json/version" 2>/dev/null | \
|
||||||
|
python3 -c "import json,sys; print(json.load(sys.stdin).get('data',{}).get('version',''))" 2>/dev/null) || pve_ver=""
|
||||||
|
if [[ -n "$pve_ver" ]]; then
|
||||||
|
echo -e " Connectivity: ${GREEN}connected (PVE ${pve_ver}) ✓${RESET}"
|
||||||
|
pve_ok=true
|
||||||
|
else
|
||||||
|
echo -e " Connectivity: ${RED}API connection failed${RESET}"
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
echo -e " Connectivity: ${YELLOW}API token not configured or PROXMOX_TOKEN_SECRET not set${RESET}"
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
if ssh -o BatchMode=yes -o ConnectTimeout=5 "${PVE_SSH_USER}@${PVE_HOST}" "true" &>/dev/null; then
|
||||||
|
echo -e " Connectivity: ${GREEN}SSH connected ✓${RESET}"
|
||||||
|
pve_ok=true
|
||||||
|
else
|
||||||
|
echo -e " Connectivity: ${RED}SSH connection failed${RESET}"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
if [[ -n "$PVE_POOL" ]]; then
|
||||||
|
if [[ "$pve_ok" == true ]] && [[ "$PVE_METHOD" == "api" ]]; then
|
||||||
|
local auth="Authorization: PVEAPIToken=${PVE_API_TOKEN_ID}=${PROXMOX_TOKEN_SECRET}"
|
||||||
|
if curl -fsSk --connect-timeout 5 -H "$auth" \
|
||||||
|
"https://${PVE_HOST}:8006/api2/json/pools/${PVE_POOL}" &>/dev/null; then
|
||||||
|
echo -e " Pool: ${GREEN}${PVE_POOL} exists ✓${RESET}"
|
||||||
|
else
|
||||||
|
echo -e " Pool: ${RED}${PVE_POOL} not found${RESET}"
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
echo -e " Pool: ${PVE_POOL} (not verified)"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
fi
|
||||||
rm -rf "$doctor_workdir"
|
rm -rf "$doctor_workdir"
|
||||||
else
|
else
|
||||||
echo -e " ${BOLD}Proxmox:${RESET} ${DIM}(requires config file, skipped)${RESET}"
|
echo -e " ${BOLD}Proxmox:${RESET} ${DIM}(no config file or proxmox.yaml found)${RESET}"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
echo ""
|
echo ""
|
||||||
|
|
@ -1106,29 +1216,121 @@ declare -a VM_VMIDS=()
|
||||||
declare -a VM_CLUSTERS=()
|
declare -a VM_CLUSTERS=()
|
||||||
declare -a VM_CLUSTER_ROLES=()
|
declare -a VM_CLUSTER_ROLES=()
|
||||||
|
|
||||||
|
# Path to the resolved proxmox connection config file (set by find_proxmox_config)
|
||||||
|
PROXMOX_CONFIG_RESOLVED=""
|
||||||
|
|
||||||
|
# Auto-discover proxmox.yaml: --proxmox flag > script directory > cwd
|
||||||
|
find_proxmox_config() {
|
||||||
|
if [[ -n "$PROXMOX_CONFIG" ]]; then
|
||||||
|
if [[ ! -f "$PROXMOX_CONFIG" ]]; then
|
||||||
|
error "Proxmox config file not found: ${PROXMOX_CONFIG}"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
PROXMOX_CONFIG_RESOLVED="$PROXMOX_CONFIG"
|
||||||
|
detail "Using proxmox config: ${PROXMOX_CONFIG_RESOLVED} (--proxmox)"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
local script_dir
|
||||||
|
script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
|
||||||
|
if [[ -f "${script_dir}/proxmox.yaml" ]]; then
|
||||||
|
PROXMOX_CONFIG_RESOLVED="${script_dir}/proxmox.yaml"
|
||||||
|
detail "Using proxmox config: ${PROXMOX_CONFIG_RESOLVED} (auto-discovered)"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ -f "proxmox.yaml" ]]; then
|
||||||
|
PROXMOX_CONFIG_RESOLVED="$(pwd)/proxmox.yaml"
|
||||||
|
detail "Using proxmox config: ${PROXMOX_CONFIG_RESOLVED} (auto-discovered)"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
# No proxmox.yaml found -- caller decides if this is an error
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
# Parse proxmox.yaml (flat key: value structure) and set PVE_* globals as base values
|
||||||
|
load_proxmox_config() {
|
||||||
|
local pxconfig="$1"
|
||||||
|
|
||||||
|
local px_json
|
||||||
|
px_json=$(mktemp "${WORKDIR}/proxmox-config-XXXXXX.json")
|
||||||
|
|
||||||
|
if ! python3 -c "
|
||||||
|
import json, sys, re
|
||||||
|
|
||||||
|
result = {}
|
||||||
|
with open(sys.argv[1]) as f:
|
||||||
|
for line in f:
|
||||||
|
s = line.strip()
|
||||||
|
if not s or s.startswith('#'):
|
||||||
|
continue
|
||||||
|
m = re.match(r'^([^:]+?):\s*(.*?)$', s)
|
||||||
|
if m:
|
||||||
|
key = m.group(1).strip()
|
||||||
|
val = m.group(2).strip()
|
||||||
|
# Strip inline comments
|
||||||
|
if val and not val.startswith(('\"', \"'\")):
|
||||||
|
cm = re.match(r'^(.*?)\s+#.*$', val)
|
||||||
|
if cm:
|
||||||
|
val = cm.group(1).rstrip()
|
||||||
|
if (val.startswith('\"') and val.endswith('\"')) or (val.startswith(\"'\") and val.endswith(\"'\")):
|
||||||
|
val = val[1:-1]
|
||||||
|
result[key] = val
|
||||||
|
|
||||||
|
json.dump(result, sys.stdout, indent=2)
|
||||||
|
" "$pxconfig" > "$px_json" 2>/dev/null; then
|
||||||
|
error "Failed to parse proxmox config: ${pxconfig}"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Set base values from proxmox.yaml (lab config overlay and CLI flags applied later)
|
||||||
|
local val
|
||||||
|
val=$(json_get "$px_json" ".host" ""); [[ -n "$val" ]] && PVE_HOST="$val"
|
||||||
|
val=$(json_get "$px_json" ".node" ""); [[ -n "$val" ]] && PVE_NODE="$val"
|
||||||
|
val=$(json_get "$px_json" ".storage" ""); [[ -n "$val" ]] && PVE_STORAGE="$val"
|
||||||
|
val=$(json_get "$px_json" ".iso_storage" ""); [[ -n "$val" ]] && PVE_ISO_STORAGE="$val"
|
||||||
|
val=$(json_get "$px_json" ".bridge" ""); [[ -n "$val" ]] && PVE_BRIDGE="$val"
|
||||||
|
val=$(json_get "$px_json" ".ssh_user" ""); [[ -n "$val" ]] && PVE_SSH_USER="$val"
|
||||||
|
val=$(json_get "$px_json" ".api_token_id" ""); [[ -n "$val" ]] && PVE_API_TOKEN_ID="$val"
|
||||||
|
val=$(json_get "$px_json" ".pool" ""); [[ -n "$val" ]] && PVE_POOL="$val"
|
||||||
|
val=$(json_get "$px_json" ".method" ""); [[ -n "$val" ]] && PVE_METHOD="$val"
|
||||||
|
|
||||||
|
rm -f "$px_json"
|
||||||
|
}
|
||||||
|
|
||||||
load_config() {
|
load_config() {
|
||||||
step "Loading configuration values"
|
step "Loading configuration values"
|
||||||
|
|
||||||
# Proxmox settings
|
# Merge priority: proxmox.yaml (base) → lab config proxmox: (overlay) → CLI flags
|
||||||
PVE_HOST=$(json_get "$CONFIG_JSON" ".proxmox.host" "")
|
# Step 1: proxmox.yaml base values were already loaded by the caller via
|
||||||
PVE_NODE=$(json_get "$CONFIG_JSON" ".proxmox.node" "pve")
|
# load_proxmox_config(). PVE_* globals may already be set.
|
||||||
PVE_STORAGE=$(json_get "$CONFIG_JSON" ".proxmox.storage" "local-lvm")
|
|
||||||
PVE_ISO_STORAGE=$(json_get "$CONFIG_JSON" ".proxmox.iso_storage" "local")
|
|
||||||
PVE_BRIDGE=$(json_get "$CONFIG_JSON" ".proxmox.bridge" "vmbr0")
|
|
||||||
PVE_SSH_USER=$(json_get "$CONFIG_JSON" ".proxmox.ssh_user" "root")
|
|
||||||
PVE_API_TOKEN_ID=$(json_get "$CONFIG_JSON" ".proxmox.api_token_id" "")
|
|
||||||
PVE_POOL=$(json_get "$CONFIG_JSON" ".proxmox.pool" "")
|
|
||||||
|
|
||||||
# Method: CLI override > config > default
|
# Step 2: overlay from lab config proxmox: section (if present)
|
||||||
if [[ -z "$PVE_METHOD" ]]; then
|
local val
|
||||||
PVE_METHOD=$(json_get "$CONFIG_JSON" ".proxmox.method" "ssh")
|
val=$(json_get "$CONFIG_JSON" ".proxmox.host" ""); [[ -n "$val" ]] && PVE_HOST="$val"
|
||||||
fi
|
val=$(json_get "$CONFIG_JSON" ".proxmox.node" ""); [[ -n "$val" ]] && PVE_NODE="$val"
|
||||||
# CLI override for method
|
val=$(json_get "$CONFIG_JSON" ".proxmox.storage" ""); [[ -n "$val" ]] && PVE_STORAGE="$val"
|
||||||
|
val=$(json_get "$CONFIG_JSON" ".proxmox.iso_storage" ""); [[ -n "$val" ]] && PVE_ISO_STORAGE="$val"
|
||||||
|
val=$(json_get "$CONFIG_JSON" ".proxmox.bridge" ""); [[ -n "$val" ]] && PVE_BRIDGE="$val"
|
||||||
|
val=$(json_get "$CONFIG_JSON" ".proxmox.ssh_user" ""); [[ -n "$val" ]] && PVE_SSH_USER="$val"
|
||||||
|
val=$(json_get "$CONFIG_JSON" ".proxmox.api_token_id" ""); [[ -n "$val" ]] && PVE_API_TOKEN_ID="$val"
|
||||||
|
val=$(json_get "$CONFIG_JSON" ".proxmox.pool" ""); [[ -n "$val" ]] && PVE_POOL="$val"
|
||||||
|
val=$(json_get "$CONFIG_JSON" ".proxmox.method" ""); [[ -n "$val" ]] && PVE_METHOD="$val"
|
||||||
|
|
||||||
|
# Step 3: apply defaults for anything still unset
|
||||||
|
[[ -z "$PVE_NODE" ]] && PVE_NODE="pve"
|
||||||
|
[[ -z "$PVE_STORAGE" ]] && PVE_STORAGE="local-lvm"
|
||||||
|
[[ -z "$PVE_ISO_STORAGE" ]] && PVE_ISO_STORAGE="local"
|
||||||
|
[[ -z "$PVE_BRIDGE" ]] && PVE_BRIDGE="vmbr0"
|
||||||
|
[[ -z "$PVE_SSH_USER" ]] && PVE_SSH_USER="root"
|
||||||
|
[[ -z "$PVE_METHOD" ]] && PVE_METHOD="ssh"
|
||||||
|
|
||||||
|
# Step 4: CLI flags override everything
|
||||||
if [[ -n "$METHOD" ]]; then
|
if [[ -n "$METHOD" ]]; then
|
||||||
PVE_METHOD="$METHOD"
|
PVE_METHOD="$METHOD"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
# Host override
|
|
||||||
if [[ -n "$HOST_OVERRIDE" ]]; then
|
if [[ -n "$HOST_OVERRIDE" ]]; then
|
||||||
PVE_HOST="$HOST_OVERRIDE"
|
PVE_HOST="$HOST_OVERRIDE"
|
||||||
fi
|
fi
|
||||||
|
|
@ -1315,7 +1517,7 @@ pve_run() {
|
||||||
local cmd="$1"
|
local cmd="$1"
|
||||||
|
|
||||||
if [[ "$DRY_RUN" == true ]]; then
|
if [[ "$DRY_RUN" == true ]]; then
|
||||||
detail "[dry-run] ${PVE_METHOD}: ${cmd}"
|
info "[dry-run] ${PVE_METHOD}: ${cmd}"
|
||||||
return 0
|
return 0
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
|
@ -1345,7 +1547,7 @@ pve_api() {
|
||||||
local auth="Authorization: PVEAPIToken=${PVE_API_TOKEN_ID}=${PROXMOX_TOKEN_SECRET:-}"
|
local auth="Authorization: PVEAPIToken=${PVE_API_TOKEN_ID}=${PROXMOX_TOKEN_SECRET:-}"
|
||||||
|
|
||||||
if [[ "$DRY_RUN" == true ]]; then
|
if [[ "$DRY_RUN" == true ]]; then
|
||||||
detail "[dry-run] API ${method} ${endpoint}"
|
info "[dry-run] API ${method} ${endpoint}"
|
||||||
if [[ -n "$data" ]]; then
|
if [[ -n "$data" ]]; then
|
||||||
detail "[dry-run] data: ${data}"
|
detail "[dry-run] data: ${data}"
|
||||||
fi
|
fi
|
||||||
|
|
@ -1387,7 +1589,7 @@ pve_upload() {
|
||||||
local remote_filename="$2"
|
local remote_filename="$2"
|
||||||
|
|
||||||
if [[ "$DRY_RUN" == true ]]; then
|
if [[ "$DRY_RUN" == true ]]; then
|
||||||
detail "[dry-run] upload: ${local_path} -> ${PVE_ISO_STORAGE}:iso/${remote_filename}"
|
info "[dry-run] upload: ${local_path} -> ${PVE_ISO_STORAGE}:iso/${remote_filename}"
|
||||||
return 0
|
return 0
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
|
@ -1529,7 +1731,7 @@ pve_create_vm() {
|
||||||
fi
|
fi
|
||||||
|
|
||||||
if [[ "$DRY_RUN" == true ]]; then
|
if [[ "$DRY_RUN" == true ]]; then
|
||||||
detail "[dry-run] ${cmd}"
|
info "[dry-run] ${cmd}"
|
||||||
return 0
|
return 0
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
|
@ -2498,6 +2700,11 @@ phase_validate() {
|
||||||
step "Phase 1: Validate"
|
step "Phase 1: Validate"
|
||||||
echo ""
|
echo ""
|
||||||
|
|
||||||
|
# Load proxmox.yaml base values (before lab config overlay)
|
||||||
|
if find_proxmox_config; then
|
||||||
|
load_proxmox_config "$PROXMOX_CONFIG_RESOLVED"
|
||||||
|
fi
|
||||||
|
|
||||||
parse_config
|
parse_config
|
||||||
load_config
|
load_config
|
||||||
validate_config
|
validate_config
|
||||||
|
|
@ -2527,7 +2734,7 @@ phase_prepare() {
|
||||||
fi
|
fi
|
||||||
elif [[ "$DRY_RUN" == true ]]; then
|
elif [[ "$DRY_RUN" == true ]]; then
|
||||||
ISO_FILENAME="IncusOS_latest.iso"
|
ISO_FILENAME="IncusOS_latest.iso"
|
||||||
detail "[dry-run] Would fetch latest version from CDN and download ISO"
|
info "[dry-run] Would fetch latest version from CDN and download ISO"
|
||||||
else
|
else
|
||||||
# Check if ISO already exists on Proxmox
|
# Check if ISO already exists on Proxmox
|
||||||
fetch_latest_version
|
fetch_latest_version
|
||||||
|
|
@ -2581,7 +2788,7 @@ phase_prepare() {
|
||||||
fi
|
fi
|
||||||
|
|
||||||
if [[ "$DRY_RUN" == true ]]; then
|
if [[ "$DRY_RUN" == true ]]; then
|
||||||
detail "[dry-run] ${seed_cmd[*]}"
|
info "[dry-run] ${seed_cmd[*]}"
|
||||||
else
|
else
|
||||||
if ! "${seed_cmd[@]}"; then
|
if ! "${seed_cmd[@]}"; then
|
||||||
error "Failed to generate seed for VM '${name}'"
|
error "Failed to generate seed for VM '${name}'"
|
||||||
|
|
@ -2708,6 +2915,78 @@ phase_create() {
|
||||||
success "All VMs created"
|
success "All VMs created"
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Remote registration
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
# Register an incus remote for a deployed VM.
|
||||||
|
# Handles stale remotes: if a remote exists but points to a different IP,
|
||||||
|
# it is removed and re-added with the correct IP (DHCP may reassign addresses
|
||||||
|
# between deploys).
|
||||||
|
# Usage: register_remote <name> <ip>
|
||||||
|
register_remote() {
|
||||||
|
local name="$1"
|
||||||
|
local ip="$2"
|
||||||
|
|
||||||
|
# Only register if incus client is available and a cert was injected
|
||||||
|
if ! command -v incus &>/dev/null; then
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
if [[ -z "$CERT_FILE_RESOLVED" ]]; then
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Check if remote already exists
|
||||||
|
local existing_url=""
|
||||||
|
existing_url=$(incus remote list --format csv 2>/dev/null | \
|
||||||
|
awk -F',' -v name="$name" '$1 == name {print $2; exit}') || true
|
||||||
|
|
||||||
|
if [[ -n "$existing_url" ]]; then
|
||||||
|
# Remote exists -- check if the IP matches
|
||||||
|
if echo "$existing_url" | grep -q "${ip}"; then
|
||||||
|
detail "Remote '${name}' already configured at ${ip}"
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Stale remote -- IP changed (DHCP reassignment between deploys)
|
||||||
|
if [[ "$DRY_RUN" == true ]]; then
|
||||||
|
info "[dry-run] Would update stale remote '${name}': ${existing_url} -> ${ip}"
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
|
||||||
|
warn "Remote '${name}' points to ${existing_url} but VM is at ${ip} -- updating"
|
||||||
|
# Switch away first -- can't remove the current/default remote
|
||||||
|
local current_remote=""
|
||||||
|
current_remote=$(incus remote get-default 2>/dev/null) || true
|
||||||
|
if [[ "$current_remote" == "$name" ]]; then
|
||||||
|
incus remote switch local 2>/dev/null || true
|
||||||
|
fi
|
||||||
|
incus remote remove "$name" 2>/dev/null || true
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ "$DRY_RUN" == true ]]; then
|
||||||
|
info "[dry-run] Would add incus remote '${name}' at ${ip}"
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Wait for port 8443 to become reachable (up to 30s)
|
||||||
|
local attempt=0
|
||||||
|
while [[ $attempt -lt 6 ]]; do
|
||||||
|
if curl -sk --connect-timeout 3 "https://${ip}:8443" &>/dev/null; then
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
sleep 5
|
||||||
|
attempt=$((attempt + 1))
|
||||||
|
done
|
||||||
|
|
||||||
|
if incus remote add "$name" "$ip" --accept-certificate --quiet 2>/dev/null; then
|
||||||
|
success "Remote '${name}' added (${ip})"
|
||||||
|
else
|
||||||
|
warn "Could not add remote '${name}' at ${ip} -- add manually:"
|
||||||
|
warn " incus remote add ${name} ${ip} --accept-certificate"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Phase 5: Install
|
# Phase 5: Install
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
@ -2730,7 +3009,7 @@ phase_install() {
|
||||||
|
|
||||||
if [[ "$DRY_RUN" == true ]]; then
|
if [[ "$DRY_RUN" == true ]]; then
|
||||||
step "Starting VM '${name}' (VMID ${vmid})"
|
step "Starting VM '${name}' (VMID ${vmid})"
|
||||||
detail "[dry-run] Would start VM, swap boot order, wait for install, clean up"
|
info "[dry-run] Would start VM, swap boot order, wait for install, clean up"
|
||||||
i=$((i + 1))
|
i=$((i + 1))
|
||||||
continue
|
continue
|
||||||
fi
|
fi
|
||||||
|
|
@ -2743,12 +3022,13 @@ phase_install() {
|
||||||
pve_vm_has_install_media "$vmid" || has_media=false
|
pve_vm_has_install_media "$vmid" || has_media=false
|
||||||
|
|
||||||
if [[ "$cur_status" == "running" ]] && [[ "$has_media" == false ]]; then
|
if [[ "$cur_status" == "running" ]] && [[ "$has_media" == false ]]; then
|
||||||
# Already installed and running — detect IP and skip
|
# Already installed and running — detect IP, register remote, skip
|
||||||
success "VM '${name}' (VMID ${vmid}) is already installed and running"
|
success "VM '${name}' (VMID ${vmid}) is already installed and running"
|
||||||
local ip
|
local ip
|
||||||
ip=$(pve_vm_ip "$vmid") || ip=""
|
ip=$(pve_vm_ip "$vmid") || ip=""
|
||||||
if [[ -n "$ip" ]]; then
|
if [[ -n "$ip" ]]; then
|
||||||
detail "IP: ${ip}"
|
detail "IP: ${ip}"
|
||||||
|
register_remote "$name" "$ip"
|
||||||
fi
|
fi
|
||||||
i=$((i + 1))
|
i=$((i + 1))
|
||||||
continue
|
continue
|
||||||
|
|
@ -2763,6 +3043,7 @@ phase_install() {
|
||||||
ip=$(pve_vm_ip "$vmid") || ip=""
|
ip=$(pve_vm_ip "$vmid") || ip=""
|
||||||
if [[ -n "$ip" ]]; then
|
if [[ -n "$ip" ]]; then
|
||||||
success "VM '${name}' started at ${ip}"
|
success "VM '${name}' started at ${ip}"
|
||||||
|
register_remote "$name" "$ip"
|
||||||
else
|
else
|
||||||
success "VM '${name}' started from disk"
|
success "VM '${name}' started from disk"
|
||||||
fi
|
fi
|
||||||
|
|
@ -2805,7 +3086,7 @@ phase_install() {
|
||||||
# installation is complete. We then stop the VM, remove install media,
|
# installation is complete. We then stop the VM, remove install media,
|
||||||
# and start cleanly from disk.
|
# and start cleanly from disk.
|
||||||
step "Waiting for '${name}' to complete installation"
|
step "Waiting for '${name}' to complete installation"
|
||||||
detail "Monitoring disk write activity (polling every 5s, timeout: 10 minutes)"
|
info "Monitoring disk write activity (polling every 5s, timeout: 10 minutes)"
|
||||||
|
|
||||||
local timeout=600
|
local timeout=600
|
||||||
local interval=5
|
local interval=5
|
||||||
|
|
@ -2840,14 +3121,14 @@ phase_install() {
|
||||||
local wr_mb=$(( wr_bytes / 1048576 ))
|
local wr_mb=$(( wr_bytes / 1048576 ))
|
||||||
# Show progress every 15s
|
# Show progress every 15s
|
||||||
if [[ $((elapsed % 15)) -eq 0 ]]; then
|
if [[ $((elapsed % 15)) -eq 0 ]]; then
|
||||||
detail "Installing... (${wr_mb} MiB written) [${elapsed}s]"
|
info "Installing... (${wr_mb} MiB written) [${elapsed}s]"
|
||||||
fi
|
fi
|
||||||
elif [[ "$writes_started" == true ]]; then
|
elif [[ "$writes_started" == true ]]; then
|
||||||
# Writes have stopped after being active
|
# Writes have stopped after being active
|
||||||
stable_count=$((stable_count + 1))
|
stable_count=$((stable_count + 1))
|
||||||
if [[ $stable_count -ge $STABLE_THRESHOLD ]]; then
|
if [[ $stable_count -ge $STABLE_THRESHOLD ]]; then
|
||||||
local wr_mb=$(( wr_bytes / 1048576 ))
|
local wr_mb=$(( wr_bytes / 1048576 ))
|
||||||
detail "Disk activity stopped (${wr_mb} MiB written total) [${elapsed}s]"
|
info "Disk activity stopped (${wr_mb} MiB written total) [${elapsed}s]"
|
||||||
install_done=true
|
install_done=true
|
||||||
break
|
break
|
||||||
fi
|
fi
|
||||||
|
|
@ -2873,7 +3154,7 @@ phase_install() {
|
||||||
local cur_status
|
local cur_status
|
||||||
cur_status=$(pve_vm_status "$vmid") || cur_status="unknown"
|
cur_status=$(pve_vm_status "$vmid") || cur_status="unknown"
|
||||||
if [[ "$cur_status" == "running" ]]; then
|
if [[ "$cur_status" == "running" ]]; then
|
||||||
detail "Stopping VM to remove install media..."
|
info "Stopping VM to remove install media..."
|
||||||
pve_stop_vm "$vmid" || true
|
pve_stop_vm "$vmid" || true
|
||||||
sleep 5
|
sleep 5
|
||||||
fi
|
fi
|
||||||
|
|
@ -2892,7 +3173,7 @@ phase_install() {
|
||||||
fi
|
fi
|
||||||
|
|
||||||
# 5. Wait for services to come up
|
# 5. Wait for services to come up
|
||||||
detail "Waiting for services to start..."
|
info "Waiting for services to start..."
|
||||||
sleep 30
|
sleep 30
|
||||||
|
|
||||||
# 6. Try to detect IP
|
# 6. Try to detect IP
|
||||||
|
|
@ -2902,12 +3183,10 @@ phase_install() {
|
||||||
echo ""
|
echo ""
|
||||||
if [[ -n "$ip" ]]; then
|
if [[ -n "$ip" ]]; then
|
||||||
success "VM '${name}' installed and running at ${ip}"
|
success "VM '${name}' installed and running at ${ip}"
|
||||||
if [[ -n "$CERT_FILE_RESOLVED" ]]; then
|
register_remote "$name" "$ip"
|
||||||
detail "Add remote: incus remote add ${name} ${ip}"
|
|
||||||
fi
|
|
||||||
else
|
else
|
||||||
success "VM '${name}' installed and running"
|
success "VM '${name}' installed and running"
|
||||||
detail "Check Proxmox console for IP address"
|
info "Check Proxmox console for IP address"
|
||||||
fi
|
fi
|
||||||
echo ""
|
echo ""
|
||||||
|
|
||||||
|
|
@ -2988,7 +3267,7 @@ phase_status() {
|
||||||
fi
|
fi
|
||||||
echo -e " Network: ${ip} port 8443: ${port_status}"
|
echo -e " Network: ${ip} port 8443: ${port_status}"
|
||||||
|
|
||||||
# Incus remote check
|
# Incus remote check + auto-register
|
||||||
if command -v incus &>/dev/null; then
|
if command -v incus &>/dev/null; then
|
||||||
local remote_name=""
|
local remote_name=""
|
||||||
remote_name=$(incus remote list --format csv 2>/dev/null | \
|
remote_name=$(incus remote list --format csv 2>/dev/null | \
|
||||||
|
|
@ -2996,7 +3275,15 @@ phase_status() {
|
||||||
if [[ -n "$remote_name" ]]; then
|
if [[ -n "$remote_name" ]]; then
|
||||||
echo -e " Remote: ${GREEN}incus remote '${remote_name}' configured${RESET}"
|
echo -e " Remote: ${GREEN}incus remote '${remote_name}' configured${RESET}"
|
||||||
else
|
else
|
||||||
echo -e " Remote: ${YELLOW}no incus remote for ${ip}${RESET}"
|
register_remote "$name" "$ip"
|
||||||
|
# Re-check after registration attempt
|
||||||
|
remote_name=$(incus remote list --format csv 2>/dev/null | \
|
||||||
|
awk -F',' -v addr="$ip" '$2 ~ addr {print $1; exit}') || true
|
||||||
|
if [[ -n "$remote_name" ]]; then
|
||||||
|
echo -e " Remote: ${GREEN}incus remote '${remote_name}' configured${RESET}"
|
||||||
|
else
|
||||||
|
echo -e " Remote: ${YELLOW}no incus remote for ${ip}${RESET}"
|
||||||
|
fi
|
||||||
fi
|
fi
|
||||||
fi
|
fi
|
||||||
else
|
else
|
||||||
|
|
@ -3032,7 +3319,7 @@ phase_post_deploy() {
|
||||||
local app="${VM_APPS[$i]}"
|
local app="${VM_APPS[$i]}"
|
||||||
|
|
||||||
if [[ "$DRY_RUN" == true ]]; then
|
if [[ "$DRY_RUN" == true ]]; then
|
||||||
detail "[dry-run] Would check '${name}' (VMID ${vmid})"
|
info "[dry-run] Would check '${name}' (VMID ${vmid})"
|
||||||
i=$((i + 1))
|
i=$((i + 1))
|
||||||
continue
|
continue
|
||||||
fi
|
fi
|
||||||
|
|
@ -3089,13 +3376,22 @@ phase_post_deploy() {
|
||||||
all_ok=false
|
all_ok=false
|
||||||
fi
|
fi
|
||||||
else
|
else
|
||||||
detail "VM '${name}': no incus remote configured"
|
# Auto-register the remote
|
||||||
detail " Add with: incus remote add ${name} ${ip} --accept-certificate"
|
register_remote "$name" "$ip"
|
||||||
|
# Verify
|
||||||
|
remote_name=$(incus remote list --format csv 2>/dev/null | \
|
||||||
|
awk -F',' -v addr="$ip" '$2 ~ addr {print $1; exit}') || true
|
||||||
|
if [[ -n "$remote_name" ]]; then
|
||||||
|
success "VM '${name}': incus remote '${remote_name}' connected"
|
||||||
|
else
|
||||||
|
info "VM '${name}': no incus remote configured"
|
||||||
|
info " Add with: incus remote add ${name} ${ip} --accept-certificate"
|
||||||
|
fi
|
||||||
fi
|
fi
|
||||||
elif [[ "$app" == "operations-center" ]]; then
|
elif [[ "$app" == "operations-center" ]]; then
|
||||||
success "VM '${name}': Operations Center at https://${ip}:8443"
|
success "VM '${name}': Operations Center at https://${ip}:8443"
|
||||||
detail " Import PKCS#12 cert in browser for web UI access:"
|
info " Import PKCS#12 cert in browser for web UI access:"
|
||||||
detail " openssl pkcs12 -export -inkey ~/.config/incus/client.key -in ~/.config/incus/client.crt -out client.pfx"
|
info " openssl pkcs12 -export -inkey ~/.config/incus/client.key -in ~/.config/incus/client.crt -out client.pfx"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
i=$((i + 1))
|
i=$((i + 1))
|
||||||
|
|
@ -3163,7 +3459,7 @@ do_cleanup() {
|
||||||
step "Destroying VM '${name}' (VMID ${vmid})"
|
step "Destroying VM '${name}' (VMID ${vmid})"
|
||||||
|
|
||||||
if [[ "$DRY_RUN" == true ]]; then
|
if [[ "$DRY_RUN" == true ]]; then
|
||||||
detail "[dry-run] Would stop and destroy VM ${vmid}"
|
info "[dry-run] Would stop and destroy VM ${vmid}"
|
||||||
destroyed=$((destroyed + 1))
|
destroyed=$((destroyed + 1))
|
||||||
i=$((i + 1))
|
i=$((i + 1))
|
||||||
continue
|
continue
|
||||||
|
|
@ -3173,7 +3469,7 @@ do_cleanup() {
|
||||||
local status
|
local status
|
||||||
status=$(pve_vm_status "$vmid") || status="unknown"
|
status=$(pve_vm_status "$vmid") || status="unknown"
|
||||||
if [[ "$status" == "running" ]]; then
|
if [[ "$status" == "running" ]]; then
|
||||||
detail "Stopping VM..."
|
info "Stopping VM..."
|
||||||
pve_stop_vm "$vmid" || true
|
pve_stop_vm "$vmid" || true
|
||||||
sleep 5
|
sleep 5
|
||||||
fi
|
fi
|
||||||
|
|
@ -3201,9 +3497,67 @@ do_cleanup() {
|
||||||
success "$summary"
|
success "$summary"
|
||||||
}
|
}
|
||||||
|
|
||||||
# Deep cleanup: remove ISOs, seeds, remotes, and local cache
|
# Detect which IncusOS ISO was used by checking ide2 config on existing VMs.
|
||||||
|
# Sets DEPLOYMENT_ISO to the filename (e.g. "IncusOS_25.03.iso") or "" if unknown.
|
||||||
|
DEPLOYMENT_ISO=""
|
||||||
|
|
||||||
|
detect_deployment_iso() {
|
||||||
|
DEPLOYMENT_ISO=""
|
||||||
|
|
||||||
|
if [[ "$DRY_RUN" == true ]]; then
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
|
||||||
|
local count=${#VM_NAMES[@]}
|
||||||
|
local i=0
|
||||||
|
while [[ $i -lt $count ]]; do
|
||||||
|
local vmid="${VM_VMIDS[$i]}"
|
||||||
|
if pve_vm_exists "$vmid"; then
|
||||||
|
local config=""
|
||||||
|
case "$PVE_METHOD" in
|
||||||
|
ssh)
|
||||||
|
config=$(pve_run "qm config ${vmid} 2>/dev/null" 2>/dev/null) || config=""
|
||||||
|
;;
|
||||||
|
api)
|
||||||
|
config=$(pve_api GET "/nodes/${PVE_NODE}/qemu/${vmid}/config" 2>/dev/null) || config=""
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
if [[ -n "$config" ]]; then
|
||||||
|
local iso_name
|
||||||
|
# Extract IncusOS ISO name from ide2 line or JSON
|
||||||
|
iso_name=$(echo "$config" | python3 -c "
|
||||||
|
import sys, json, re
|
||||||
|
text = sys.stdin.read()
|
||||||
|
# Try JSON (API response)
|
||||||
|
try:
|
||||||
|
data = json.loads(text)
|
||||||
|
ide2 = data.get('data', data).get('ide2', '')
|
||||||
|
except (json.JSONDecodeError, AttributeError):
|
||||||
|
ide2 = ''
|
||||||
|
for line in text.splitlines():
|
||||||
|
if line.startswith('ide2:'):
|
||||||
|
ide2 = line.split(':', 1)[1].strip()
|
||||||
|
break
|
||||||
|
m = re.search(r'(IncusOS_[^,]+\.iso)', ide2)
|
||||||
|
if m:
|
||||||
|
print(m.group(1))
|
||||||
|
" 2>/dev/null) || iso_name=""
|
||||||
|
|
||||||
|
if [[ -n "$iso_name" ]]; then
|
||||||
|
DEPLOYMENT_ISO="$iso_name"
|
||||||
|
detail "Detected deployment ISO: ${DEPLOYMENT_ISO} (from VM ${vmid})"
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
i=$((i + 1))
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
# Deep cleanup: remove seeds and the specific IncusOS ISO used by this deployment
|
||||||
cleanup_storage() {
|
cleanup_storage() {
|
||||||
step "Deep cleanup: removing ISOs and seeds from Proxmox storage"
|
step "Deep cleanup: removing seeds and deployment ISO from Proxmox storage"
|
||||||
|
|
||||||
local count=${#VM_NAMES[@]}
|
local count=${#VM_NAMES[@]}
|
||||||
local i=0
|
local i=0
|
||||||
|
|
@ -3218,7 +3572,7 @@ cleanup_storage() {
|
||||||
|
|
||||||
local seed_file="seed-${name}.iso"
|
local seed_file="seed-${name}.iso"
|
||||||
if [[ "$DRY_RUN" == true ]]; then
|
if [[ "$DRY_RUN" == true ]]; then
|
||||||
detail "[dry-run] Would delete ${PVE_ISO_STORAGE}:iso/${seed_file}"
|
info "[dry-run] Would delete ${PVE_ISO_STORAGE}:iso/${seed_file}"
|
||||||
else
|
else
|
||||||
case "$PVE_METHOD" in
|
case "$PVE_METHOD" in
|
||||||
ssh)
|
ssh)
|
||||||
|
|
@ -3228,18 +3582,46 @@ cleanup_storage() {
|
||||||
pve_api DELETE "/nodes/${PVE_NODE}/storage/${PVE_ISO_STORAGE}/content/${PVE_ISO_STORAGE}:iso/${seed_file}" &>/dev/null || true
|
pve_api DELETE "/nodes/${PVE_NODE}/storage/${PVE_ISO_STORAGE}/content/${PVE_ISO_STORAGE}:iso/${seed_file}" &>/dev/null || true
|
||||||
;;
|
;;
|
||||||
esac
|
esac
|
||||||
detail "Deleted seed: ${seed_file}"
|
info "Deleted seed: ${seed_file}"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
i=$((i + 1))
|
i=$((i + 1))
|
||||||
done
|
done
|
||||||
|
|
||||||
# Remove IncusOS ISOs from Proxmox storage
|
# Remove only the specific IncusOS ISO used by this deployment
|
||||||
|
if [[ -n "$DEPLOYMENT_ISO" ]]; then
|
||||||
|
local volid="${PVE_ISO_STORAGE}:iso/${DEPLOYMENT_ISO}"
|
||||||
|
if [[ "$DRY_RUN" == true ]]; then
|
||||||
|
info "[dry-run] Would delete ${volid}"
|
||||||
|
else
|
||||||
|
case "$PVE_METHOD" in
|
||||||
|
ssh)
|
||||||
|
pve_run "pvesm free ${volid} 2>/dev/null" &>/dev/null || true
|
||||||
|
;;
|
||||||
|
api)
|
||||||
|
local encoded_volid
|
||||||
|
encoded_volid=$(python3 -c "import urllib.parse; print(urllib.parse.quote('${volid}', safe=''))" 2>/dev/null)
|
||||||
|
pve_api DELETE "/nodes/${PVE_NODE}/storage/${PVE_ISO_STORAGE}/content/${encoded_volid}" &>/dev/null || true
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
info "Deleted ISO: ${DEPLOYMENT_ISO}"
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
detail "No deployment ISO detected (VMs may already be destroyed)"
|
||||||
|
fi
|
||||||
|
|
||||||
|
success "Storage cleanup complete"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Aggressive cleanup: remove ALL IncusOS ISOs and seed ISOs from storage
|
||||||
|
cleanup_storage_all() {
|
||||||
|
step "Deep cleanup: removing ALL IncusOS ISOs and seeds from Proxmox storage"
|
||||||
|
|
||||||
local iso_list=""
|
local iso_list=""
|
||||||
if [[ "$DRY_RUN" != true ]]; then
|
if [[ "$DRY_RUN" != true ]]; then
|
||||||
case "$PVE_METHOD" in
|
case "$PVE_METHOD" in
|
||||||
ssh)
|
ssh)
|
||||||
iso_list=$(pve_run "pvesm list ${PVE_ISO_STORAGE} --content iso 2>/dev/null" 2>/dev/null | grep "IncusOS_" | awk '{print $1}') || iso_list=""
|
iso_list=$(pve_run "pvesm list ${PVE_ISO_STORAGE} --content iso 2>/dev/null" 2>/dev/null | grep -E "(IncusOS_|seed-)" | awk '{print $1}') || iso_list=""
|
||||||
;;
|
;;
|
||||||
api)
|
api)
|
||||||
iso_list=$(pve_api GET "/nodes/${PVE_NODE}/storage/${PVE_ISO_STORAGE}/content?content=iso" 2>/dev/null | \
|
iso_list=$(pve_api GET "/nodes/${PVE_NODE}/storage/${PVE_ISO_STORAGE}/content?content=iso" 2>/dev/null | \
|
||||||
|
|
@ -3248,7 +3630,7 @@ import json, sys
|
||||||
data = json.load(sys.stdin)
|
data = json.load(sys.stdin)
|
||||||
for item in data.get('data', []):
|
for item in data.get('data', []):
|
||||||
volid = item.get('volid', '')
|
volid = item.get('volid', '')
|
||||||
if 'IncusOS_' in volid:
|
if 'IncusOS_' in volid or 'seed-' in volid:
|
||||||
print(volid)
|
print(volid)
|
||||||
" 2>/dev/null) || iso_list=""
|
" 2>/dev/null) || iso_list=""
|
||||||
;;
|
;;
|
||||||
|
|
@ -3259,7 +3641,7 @@ for item in data.get('data', []):
|
||||||
while IFS= read -r volid; do
|
while IFS= read -r volid; do
|
||||||
[[ -z "$volid" ]] && continue
|
[[ -z "$volid" ]] && continue
|
||||||
if [[ "$DRY_RUN" == true ]]; then
|
if [[ "$DRY_RUN" == true ]]; then
|
||||||
detail "[dry-run] Would delete ${volid}"
|
info "[dry-run] Would delete ${volid}"
|
||||||
else
|
else
|
||||||
case "$PVE_METHOD" in
|
case "$PVE_METHOD" in
|
||||||
ssh)
|
ssh)
|
||||||
|
|
@ -3271,14 +3653,18 @@ for item in data.get('data', []):
|
||||||
pve_api DELETE "/nodes/${PVE_NODE}/storage/${PVE_ISO_STORAGE}/content/${encoded_volid}" &>/dev/null || true
|
pve_api DELETE "/nodes/${PVE_NODE}/storage/${PVE_ISO_STORAGE}/content/${encoded_volid}" &>/dev/null || true
|
||||||
;;
|
;;
|
||||||
esac
|
esac
|
||||||
detail "Deleted ISO: ${volid}"
|
info "Deleted: ${volid}"
|
||||||
fi
|
fi
|
||||||
done <<< "$iso_list"
|
done <<< "$iso_list"
|
||||||
|
else
|
||||||
|
info "No IncusOS ISOs or seeds found in storage"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
success "Storage cleanup complete"
|
success "Storage cleanup complete"
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# Remove incus remotes. Uses VM_NAMES (config-based cleanup) or
|
||||||
|
# DESTROYED_VM_NAMES (pool-wide cleanup) depending on what's populated.
|
||||||
cleanup_remotes() {
|
cleanup_remotes() {
|
||||||
step "Deep cleanup: removing incus remotes"
|
step "Deep cleanup: removing incus remotes"
|
||||||
|
|
||||||
|
|
@ -3287,27 +3673,40 @@ cleanup_remotes() {
|
||||||
return
|
return
|
||||||
fi
|
fi
|
||||||
|
|
||||||
local count=${#VM_NAMES[@]}
|
# Determine which name list to use
|
||||||
local i=0
|
local names_to_check=()
|
||||||
|
if [[ ${#DESTROYED_VM_NAMES[@]} -gt 0 ]]; then
|
||||||
while [[ $i -lt $count ]]; do
|
# Pool-wide cleanup: use names of VMs actually destroyed
|
||||||
local name="${VM_NAMES[$i]}"
|
names_to_check=("${DESTROYED_VM_NAMES[@]}")
|
||||||
if [[ -n "$VM_FILTER" ]] && [[ "$name" != "$VM_FILTER" ]]; then
|
elif [[ ${#VM_NAMES[@]} -gt 0 ]]; then
|
||||||
|
# Config-based cleanup
|
||||||
|
local i=0
|
||||||
|
while [[ $i -lt ${#VM_NAMES[@]} ]]; do
|
||||||
|
if [[ -n "$VM_FILTER" ]] && [[ "${VM_NAMES[$i]}" != "$VM_FILTER" ]]; then
|
||||||
|
i=$((i + 1))
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
names_to_check+=("${VM_NAMES[$i]}")
|
||||||
i=$((i + 1))
|
i=$((i + 1))
|
||||||
continue
|
done
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
if [[ ${#names_to_check[@]} -eq 0 ]]; then
|
||||||
|
detail "No VM names to check for remote cleanup"
|
||||||
|
success "Remote cleanup complete"
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
|
||||||
|
for name in "${names_to_check[@]}"; do
|
||||||
# Check if a remote with this name exists
|
# Check if a remote with this name exists
|
||||||
if incus remote list --format csv 2>/dev/null | grep -q "^${name},"; then
|
if incus remote list --format csv 2>/dev/null | grep -q "^${name},"; then
|
||||||
if [[ "$DRY_RUN" == true ]]; then
|
if [[ "$DRY_RUN" == true ]]; then
|
||||||
detail "[dry-run] Would remove incus remote '${name}'"
|
info "[dry-run] Would remove incus remote '${name}'"
|
||||||
else
|
else
|
||||||
incus remote remove "$name" 2>/dev/null || true
|
incus remote remove "$name" 2>/dev/null || true
|
||||||
detail "Removed incus remote: ${name}"
|
info "Removed incus remote: ${name}"
|
||||||
fi
|
fi
|
||||||
fi
|
fi
|
||||||
|
|
||||||
i=$((i + 1))
|
|
||||||
done
|
done
|
||||||
|
|
||||||
success "Remote cleanup complete"
|
success "Remote cleanup complete"
|
||||||
|
|
@ -3319,10 +3718,10 @@ cleanup_local_cache() {
|
||||||
local cache_base="${XDG_CACHE_HOME:-${HOME}/.cache}/${SCRIPT_NAME}/iso-cache"
|
local cache_base="${XDG_CACHE_HOME:-${HOME}/.cache}/${SCRIPT_NAME}/iso-cache"
|
||||||
if [[ -d "$cache_base" ]]; then
|
if [[ -d "$cache_base" ]]; then
|
||||||
if [[ "$DRY_RUN" == true ]]; then
|
if [[ "$DRY_RUN" == true ]]; then
|
||||||
detail "[dry-run] Would clear ${cache_base}"
|
info "[dry-run] Would clear ${cache_base}"
|
||||||
else
|
else
|
||||||
rm -rf "$cache_base"
|
rm -rf "$cache_base"
|
||||||
detail "Cleared cache: ${cache_base}"
|
info "Cleared cache: ${cache_base}"
|
||||||
fi
|
fi
|
||||||
else
|
else
|
||||||
detail "No local cache to clear"
|
detail "No local cache to clear"
|
||||||
|
|
@ -3331,13 +3730,17 @@ cleanup_local_cache() {
|
||||||
success "Local cache cleanup complete"
|
success "Local cache cleanup complete"
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# Names of VMs destroyed during cleanup-all (used by cleanup_remotes)
|
||||||
|
declare -a DESTROYED_VM_NAMES=()
|
||||||
|
|
||||||
# Pool-wide cleanup: destroy ALL managed VMs in the resource pool
|
# Pool-wide cleanup: destroy ALL managed VMs in the resource pool
|
||||||
do_cleanup_all() {
|
do_cleanup_all() {
|
||||||
step "Pool-wide cleanup: destroying all managed VMs"
|
step "Pool-wide cleanup: destroying all managed VMs"
|
||||||
echo ""
|
echo ""
|
||||||
|
|
||||||
if [[ -z "$PVE_POOL" ]]; then
|
if [[ -z "$PVE_POOL" ]]; then
|
||||||
error "Pool-wide cleanup requires a resource pool (proxmox.pool) in the config"
|
error "Pool-wide cleanup requires a resource pool (proxmox.pool)"
|
||||||
|
error "Set pool in proxmox.yaml or pass a config with proxmox.pool"
|
||||||
error "Without a pool, there is no safe way to scope which VMs to destroy"
|
error "Without a pool, there is no safe way to scope which VMs to destroy"
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
|
@ -3389,8 +3792,9 @@ do_cleanup_all() {
|
||||||
step "Destroying VM '${name}' (VMID ${vmid})"
|
step "Destroying VM '${name}' (VMID ${vmid})"
|
||||||
|
|
||||||
if [[ "$DRY_RUN" == true ]]; then
|
if [[ "$DRY_RUN" == true ]]; then
|
||||||
detail "[dry-run] Would stop and destroy VM ${vmid}"
|
info "[dry-run] Would stop and destroy VM ${vmid}"
|
||||||
destroyed=$((destroyed + 1))
|
destroyed=$((destroyed + 1))
|
||||||
|
DESTROYED_VM_NAMES+=("$name")
|
||||||
idx=$((idx + 1))
|
idx=$((idx + 1))
|
||||||
continue
|
continue
|
||||||
fi
|
fi
|
||||||
|
|
@ -3398,7 +3802,7 @@ do_cleanup_all() {
|
||||||
local status
|
local status
|
||||||
status=$(pve_vm_status "$vmid") || status="unknown"
|
status=$(pve_vm_status "$vmid") || status="unknown"
|
||||||
if [[ "$status" == "running" ]]; then
|
if [[ "$status" == "running" ]]; then
|
||||||
detail "Stopping VM..."
|
info "Stopping VM..."
|
||||||
pve_stop_vm "$vmid" || true
|
pve_stop_vm "$vmid" || true
|
||||||
sleep 5
|
sleep 5
|
||||||
fi
|
fi
|
||||||
|
|
@ -3406,6 +3810,7 @@ do_cleanup_all() {
|
||||||
if pve_destroy_vm "$vmid"; then
|
if pve_destroy_vm "$vmid"; then
|
||||||
success "VM '${name}' (VMID ${vmid}) destroyed"
|
success "VM '${name}' (VMID ${vmid}) destroyed"
|
||||||
destroyed=$((destroyed + 1))
|
destroyed=$((destroyed + 1))
|
||||||
|
DESTROYED_VM_NAMES+=("$name")
|
||||||
else
|
else
|
||||||
error "Failed to destroy VM '${name}' (VMID ${vmid})"
|
error "Failed to destroy VM '${name}' (VMID ${vmid})"
|
||||||
fi
|
fi
|
||||||
|
|
@ -3546,6 +3951,40 @@ main() {
|
||||||
|
|
||||||
setup_workdir
|
setup_workdir
|
||||||
|
|
||||||
|
# Handle cleanup-all without a config file (only needs proxmox.yaml)
|
||||||
|
if [[ "$CLEANUP_ALL" == true ]] && [[ -z "$CONFIG_FILE" ]]; then
|
||||||
|
if ! find_proxmox_config; then
|
||||||
|
error "No proxmox.yaml found and no config file specified"
|
||||||
|
error "Create proxmox.yaml or use --proxmox FILE"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
load_proxmox_config "$PROXMOX_CONFIG_RESOLVED"
|
||||||
|
|
||||||
|
# Apply defaults for anything still unset
|
||||||
|
[[ -z "$PVE_NODE" ]] && PVE_NODE="pve"
|
||||||
|
[[ -z "$PVE_STORAGE" ]] && PVE_STORAGE="local-lvm"
|
||||||
|
[[ -z "$PVE_ISO_STORAGE" ]] && PVE_ISO_STORAGE="local"
|
||||||
|
[[ -z "$PVE_BRIDGE" ]] && PVE_BRIDGE="vmbr0"
|
||||||
|
[[ -z "$PVE_SSH_USER" ]] && PVE_SSH_USER="root"
|
||||||
|
[[ -z "$PVE_METHOD" ]] && PVE_METHOD="ssh"
|
||||||
|
[[ -n "$METHOD" ]] && PVE_METHOD="$METHOD"
|
||||||
|
[[ -n "$HOST_OVERRIDE" ]] && PVE_HOST="$HOST_OVERRIDE"
|
||||||
|
|
||||||
|
if [[ -z "$PVE_HOST" ]]; then
|
||||||
|
error "proxmox.yaml must contain 'host:'"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
do_cleanup_all
|
||||||
|
if [[ "$DEEP_CLEANUP" == true ]]; then
|
||||||
|
echo ""
|
||||||
|
cleanup_storage_all
|
||||||
|
cleanup_remotes
|
||||||
|
cleanup_local_cache
|
||||||
|
fi
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
|
||||||
if [[ "$DRY_RUN" != true ]]; then
|
if [[ "$DRY_RUN" != true ]]; then
|
||||||
check_dependencies
|
check_dependencies
|
||||||
fi
|
fi
|
||||||
|
|
@ -3554,12 +3993,12 @@ main() {
|
||||||
# Always run validate first
|
# Always run validate first
|
||||||
phase_validate
|
phase_validate
|
||||||
|
|
||||||
# Handle cleanup-all mode (pool-wide, not config-specific VMs)
|
# Handle cleanup-all mode (with config file for additional context)
|
||||||
if [[ "$CLEANUP_ALL" == true ]]; then
|
if [[ "$CLEANUP_ALL" == true ]]; then
|
||||||
do_cleanup_all
|
do_cleanup_all
|
||||||
if [[ "$DEEP_CLEANUP" == true ]]; then
|
if [[ "$DEEP_CLEANUP" == true ]]; then
|
||||||
echo ""
|
echo ""
|
||||||
cleanup_storage
|
cleanup_storage_all
|
||||||
cleanup_remotes
|
cleanup_remotes
|
||||||
cleanup_local_cache
|
cleanup_local_cache
|
||||||
fi
|
fi
|
||||||
|
|
@ -3568,6 +4007,10 @@ main() {
|
||||||
|
|
||||||
# Handle cleanup mode
|
# Handle cleanup mode
|
||||||
if [[ "$CLEANUP" == true ]]; then
|
if [[ "$CLEANUP" == true ]]; then
|
||||||
|
if [[ "$DEEP_CLEANUP" == true ]]; then
|
||||||
|
# Detect which ISO was used before destroying VMs
|
||||||
|
detect_deployment_iso
|
||||||
|
fi
|
||||||
confirm "Destroy VMs listed above?"
|
confirm "Destroy VMs listed above?"
|
||||||
do_cleanup
|
do_cleanup
|
||||||
if [[ "$DEEP_CLEANUP" == true ]]; then
|
if [[ "$DEEP_CLEANUP" == true ]]; then
|
||||||
|
|
|
||||||
File diff suppressed because it is too large
Load Diff
Loading…
Reference in New Issue