98 lines
3.4 KiB
Markdown
98 lines
3.4 KiB
Markdown
---
|
|
paths:
|
|
- "incusos/lab-test"
|
|
- "incusos/incusos-proxmox"
|
|
- "notes/clustering-guide.md"
|
|
- "notes/production-lab-guide.md"
|
|
---
|
|
|
|
# Incus Clustering
|
|
|
|
## Overview
|
|
|
|
- Cluster formation via `incus` CLI using remotes. No SSH needed (IncusOS is immutable).
|
|
- No VIP needed: each node advertises its own IP. Requests forwarded internally.
|
|
|
|
## Pre-clustering: fix core.https_address
|
|
|
|
- Default `:8443` (wildcard). Clustering needs specific routable IP.
|
|
- Set on every node BEFORE clustering:
|
|
```bash
|
|
incus config set <remote>: core.https_address <NODE_IP>:8443
|
|
```
|
|
- Get routable IP:
|
|
```bash
|
|
incus query <remote>:/1.0 | python3 -c "import sys,json; d=json.load(sys.stdin); \
|
|
[print(a) for a in d['environment']['addresses'] \
|
|
if not a.startswith('10.') and not a.startswith('fd42:') and not a.startswith('[')]"
|
|
```
|
|
|
|
## Cluster enable (init node)
|
|
|
|
- ```bash
|
|
incus cluster enable <remote>: <member-name>
|
|
```
|
|
TWO arguments: `<remote>:` (trailing colon) and `<member-name>`.
|
|
- Generates new TLS cert. Fix remote:
|
|
```bash
|
|
incus remote switch local
|
|
incus remote remove <remote>
|
|
incus remote add <remote> https://<NODE_IP>:8443 --accept-certificate
|
|
```
|
|
|
|
## apply_defaults for clusters
|
|
|
|
- **Bootstrap node**: `apply_defaults: true` (needs pool + network)
|
|
- **Joining nodes**: `apply_defaults: false` (recommended, clean join)
|
|
- **Standalone**: `apply_defaults: true`
|
|
- If `apply_defaults: true` on joiners, 8-command cleanup needed before join
|
|
(delete pool, network, volumes, profile devices).
|
|
|
|
## Join workflow
|
|
|
|
- **Token**: `incus cluster add <init-remote>:<new-member-name>` (ONE arg)
|
|
- **Join**: `incus cluster join <init-remote>: <joining-remote>:` (TWO args)
|
|
- **Automated**:
|
|
```bash
|
|
printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join <init-remote>: <joining-remote>:
|
|
```
|
|
- After join: fix remote (same as init).
|
|
|
|
## Command syntax gotchas
|
|
|
|
- `cluster enable remote: member-name` -- TWO args
|
|
- `cluster add remote:member-name` -- ONE arg
|
|
- `cluster remove remote:member-name --force` -- ONE arg, prompts "yes/no"
|
|
- `cluster evacuate remote:member-name` -- ONE arg
|
|
- `cluster restore remote:member-name` -- ONE arg
|
|
- `cluster join init-remote: joining-remote:` -- TWO args
|
|
- `config set remote: key value` -- remote with trailing colon + space
|
|
- General: `remote:resource` for resource, `remote:` for server itself
|
|
|
|
## Workload placement and migration
|
|
|
|
- **Targeted launch**: `incus launch images:debian/12 <remote>:name --target <member>`
|
|
- **Container migration**: stop/move/start only (no CRIU).
|
|
- **VM live migration**: requires `migration.stateful=true` (set while stopped).
|
|
- **limits.cpu MUST be a range** (e.g., `0-1`), not integer. Integer causes
|
|
`maxcpus` mismatch -> `Missing section footer for ICH9LPC`.
|
|
- **size.state required**: `size.state=2GiB` on root disk for stateful ops.
|
|
- **Agent reconnect**: sleep 4s after migration before `incus exec`.
|
|
- **Evacuation**: `incus cluster evacuate <remote>:<member> --force`
|
|
Use `--action stop` without limits.cpu range fix.
|
|
|
|
## Cluster rebalancing
|
|
|
|
```bash
|
|
incus config set <remote>: cluster.rebalance.interval=1
|
|
incus config set <remote>: cluster.rebalance.threshold=10
|
|
incus config set <remote>: cluster.rebalance.batch=2
|
|
incus config set <remote>: cluster.rebalance.cooldown=5m
|
|
```
|
|
|
|
## Lab validation (lab-test)
|
|
|
|
- Reads same YAML config as `incusos-proxmox`.
|
|
- Phases: deploy, single, cluster, workload, migrate.
|
|
- Test instances: `test-*` prefix. Reports PASS/FAIL/SKIP.
|