1739 lines
48 KiB
Markdown
1739 lines
48 KiB
Markdown
# Production Home Lab Guide
|
|
|
|
Build a production-quality Incus home lab from scratch: Operations Center
|
|
dashboard, 3-node cluster with OVN overlay networking, mixed container/VM
|
|
workloads, live migration, network security, load balancers, and cluster
|
|
lifecycle management.
|
|
|
|
All commands and output in this guide are from an actual deployment on
|
|
2026-02-22. Tested on Proxmox VE 9.1.5, IncusOS 202602210344, Incus
|
|
client 6.21, Operations Center v0.3.0.
|
|
|
|
|
|
## Section 0: Architecture Overview
|
|
|
|
### Network Topology
|
|
|
|
```mermaid
|
|
flowchart TD
|
|
vlan(("VLAN 69<br/>192.168.100.0/22"))
|
|
|
|
subgraph mgmt["Management"]
|
|
oc["lab-oc<br/>VMID 910 · .110<br/>OC server"]
|
|
end
|
|
|
|
subgraph cluster["Incus Cluster"]
|
|
n1["lab-node-01<br/>VMID 911 · .111<br/>init + ovn-central"]
|
|
n2["lab-node-02<br/>VMID 912 · .112"]
|
|
n3["lab-node-03<br/>VMID 913 · .113"]
|
|
end
|
|
|
|
subgraph networks["OVN Networks"]
|
|
prod("net-prod<br/>10.10.10.0/24")
|
|
iso("net-isolated<br/>10.10.20.0/24")
|
|
uplink("UPLINK<br/>.103.200-210")
|
|
end
|
|
|
|
vlan --- mgmt & cluster
|
|
cluster -->|"Geneve tunnels"| networks
|
|
|
|
classDef nodeClass fill:#009E73,color:#fff,stroke:#007a5e
|
|
classDef mgmtClass fill:#CC79A7,color:#fff,stroke:#a36088
|
|
classDef networkClass fill:#0072B2,color:#fff,stroke:#005a8e
|
|
|
|
class n1,n2,n3 nodeClass
|
|
class oc mgmtClass
|
|
class prod,iso,uplink,vlan networkClass
|
|
|
|
style mgmt fill:#f5e6f0,stroke:#CC79A7
|
|
style cluster fill:#e6f5f0,stroke:#009E73
|
|
style networks fill:#e0eef8,stroke:#0072B2
|
|
```
|
|
|
|
### Infrastructure
|
|
|
|
| Component | VMID | IP | Cores | RAM | Disk | Role |
|
|
|-----------|------|-----|-------|-----|------|------|
|
|
| lab-oc | 910 | 192.168.102.110/22 | 2 | 4 GiB | 50G | Operations Center |
|
|
| lab-node-01 | 911 | 192.168.102.111/22 | 4 | 8 GiB | 64G | Cluster init + OVN host |
|
|
| lab-node-02 | 912 | 192.168.102.112/22 | 4 | 8 GiB | 50G | Cluster member |
|
|
| lab-node-03 | 913 | 192.168.102.113/22 | 4 | 8 GiB | 50G | Cluster member |
|
|
|
|
**RAM budget**: 28 GiB of 64 GiB (44% utilization). Leaves headroom for
|
|
workloads inside the VMs and other labs on the host.
|
|
|
|
**OVN IP allocation**: 192.168.103.200-210 reserved for OVN external
|
|
addresses (router IPs, load balancer VIPs, network forwards). These must
|
|
be excluded from your DHCP server's range.
|
|
|
|
### Decision Rationale
|
|
|
|
**Why manual clustering instead of OC `provisioning cluster add`?**
|
|
OC v0.3.0's `provisioning cluster add` has a `needs update: false` blocker
|
|
that can stall indefinitely. Manual clustering via `incus` CLI is proven
|
|
reliable. OC still provides value as a monitoring dashboard.
|
|
|
|
**Why OVN?** Bridge networks are node-local — instances on different nodes
|
|
cannot communicate. OVN provides transparent cross-node L2 overlay with
|
|
sub-millisecond latency (~0.1-0.8ms), network isolation, ACLs, load
|
|
balancers, and network forwards.
|
|
|
|
**Why VLAN 69?** Isolates lab traffic from the production LAN. All VMs share
|
|
VLAN 69 (subnet 192.168.100.0/22). The VLAN tag is set at the Proxmox VM
|
|
level — IncusOS and workloads are unaware of it.
|
|
|
|
### Cross-References
|
|
|
|
This guide brings together techniques from the deep-dive guides:
|
|
|
|
- [Clustering Guide](clustering-guide.md) — cluster formation, migration, lifecycle
|
|
- [Networking Guide](networking-guide.md) — OVN setup, ACLs, peering, LBs
|
|
- [Operations Center Guide](operations-center-guide.md) — OC provisioning, CLI, web UI
|
|
- [Migration Guide](migration-guide.md) — importing VMs from other hypervisors
|
|
|
|
|
|
## Section 1: Prerequisites
|
|
|
|
### Required Tools
|
|
|
|
Verify all tools are available before starting:
|
|
|
|
```bash
|
|
incus version
|
|
operations-center --version
|
|
bash --version | head -1
|
|
python3 --version
|
|
jq --version
|
|
curl --version | head -1
|
|
genisoimage --version 2>&1 | head -1
|
|
```
|
|
|
|
**Minimum versions**: Incus client 6.3+ (for `remote get-client-certificate`
|
|
fallback, though scripts read `~/.config/incus/client.crt` directly),
|
|
Operations Center v0.3.0+.
|
|
|
|
### Proxmox Configuration
|
|
|
|
Your `incusos/proxmox.yaml` should contain:
|
|
|
|
```yaml
|
|
host: 192.168.1.29
|
|
method: api
|
|
api_token_id: automation@pve!deploy
|
|
node: pve
|
|
storage: local-zfs
|
|
iso_storage: local
|
|
bridge: vmbr0
|
|
vlan: 69
|
|
gateway: 192.168.100.1
|
|
dns: 192.168.100.1
|
|
pool: IncusLab
|
|
```
|
|
|
|
The `env` file at the repository root must export `PROXMOX_TOKEN_SECRET`.
|
|
Scripts auto-discover it — no manual sourcing needed.
|
|
|
|
### Client Certificates
|
|
|
|
Incus client certificates are used for both Incus and OC connections:
|
|
|
|
```bash
|
|
# Verify cert exists (auto-generated on first incus command)
|
|
ls -la ~/.config/incus/client.crt ~/.config/incus/client.key
|
|
```
|
|
|
|
For OC web UI browser access, generate a PKCS#12 bundle:
|
|
|
|
```bash
|
|
openssl pkcs12 -export \
|
|
-out ~/.config/incus/client.pfx \
|
|
-inkey ~/.config/incus/client.key \
|
|
-in ~/.config/incus/client.crt \
|
|
-name "Incus Client"
|
|
```
|
|
|
|
Import `client.pfx` into your browser's certificate store (Firefox:
|
|
Settings → Privacy & Security → View Certificates → Import).
|
|
|
|
### Doctor Check
|
|
|
|
Run the environment check to verify everything is in order:
|
|
|
|
```bash
|
|
cd incusos
|
|
./incusos-proxmox --doctor
|
|
```
|
|
|
|
Expected output includes tool versions, IncusOS CDN reachability,
|
|
`proxmox.yaml` discovery, and Proxmox API connectivity.
|
|
|
|
|
|
## Section 2: Deploy Infrastructure
|
|
|
|
### Configuration File
|
|
|
|
The lab uses `incusos/examples/lab-production.yaml`:
|
|
|
|
```yaml
|
|
defaults:
|
|
cores: 4
|
|
memory: 8192
|
|
disk: 50
|
|
start_vmid: 910
|
|
|
|
vms:
|
|
- name: lab-oc
|
|
app: operations-center
|
|
apply_defaults: true
|
|
cores: 2
|
|
memory: 4096
|
|
ip: 192.168.102.110/22
|
|
|
|
- name: lab-node-01
|
|
app: incus
|
|
apply_defaults: true # init node: needs storage pool + network
|
|
disk: 64 # extra space for OVN control plane container
|
|
ip: 192.168.102.111/22
|
|
|
|
- name: lab-node-02
|
|
app: incus
|
|
apply_defaults: false # joining node: cluster join creates pool entry
|
|
ip: 192.168.102.112/22
|
|
|
|
- name: lab-node-03
|
|
app: incus
|
|
apply_defaults: false # joining node: cluster join creates pool entry
|
|
ip: 192.168.102.113/22
|
|
```
|
|
|
|
**Key decisions**: node-01 has `apply_defaults: true` (cluster init needs
|
|
storage pool and network bridge). Nodes 02 and 03 have `apply_defaults: false`
|
|
(the cluster join process creates member-specific entries). node-01 gets
|
|
64 GiB disk for the OVN control plane container.
|
|
|
|
### Dry Run
|
|
|
|
Preview the deployment without making any changes:
|
|
|
|
```bash
|
|
./incusos-proxmox --dry-run examples/lab-production.yaml
|
|
```
|
|
|
|
This shows: ISO download plan, seed generation commands, VM creation
|
|
parameters, and the full install sequence for each VM.
|
|
|
|
### Deploy
|
|
|
|
Deploy all 4 VMs:
|
|
|
|
```bash
|
|
./incusos-proxmox --yes examples/lab-production.yaml
|
|
```
|
|
|
|
The deploy takes ~5-8 minutes:
|
|
|
|
1. Downloads the latest IncusOS ISO (if not cached)
|
|
2. Generates per-VM seed ISOs with static IP, hostname, certificates
|
|
3. Creates VMs on Proxmox with UEFI, TPM, VirtIO settings
|
|
4. Boots each VM with ISO + seed, monitors installation via blockstat
|
|
5. Detects install completion (876 MiB written, then idle)
|
|
6. Stops VMs, removes install media (ISO + seed)
|
|
7. Starts VMs from disk, waits for port 8443 (up to 180s)
|
|
8. Auto-heals scrub_schedule via IncusOS REST API
|
|
9. Configures `incus` remotes for each Incus node
|
|
|
|
### Verify Deployment
|
|
|
|
Check deployment status:
|
|
|
|
```bash
|
|
./incusos-proxmox --status examples/lab-production.yaml
|
|
```
|
|
|
|
Expected output shows each VM with Proxmox state (running), network
|
|
(static IP reachable), port 8443 (open), and incus remote (configured).
|
|
|
|
### Verify Scrub Schedule
|
|
|
|
Confirm the crontab bug fix is effective on all Incus nodes:
|
|
|
|
```bash
|
|
for node in lab-node-01 lab-node-02 lab-node-03; do
|
|
echo -n "$node scrub_schedule: "
|
|
incus query "$node":/os/1.0/system/storage | python3 -c \
|
|
"import sys,json; print(json.load(sys.stdin).get('config',{}).get('scrub_schedule','EMPTY'))"
|
|
done
|
|
```
|
|
|
|
Expected output — all nodes show `0 4 * * 0`:
|
|
|
|
```
|
|
lab-node-01 scrub_schedule: 0 4 * * 0
|
|
lab-node-02 scrub_schedule: 0 4 * * 0
|
|
lab-node-03 scrub_schedule: 0 4 * * 0
|
|
```
|
|
|
|
If any node shows `EMPTY`, the crontab bug hit. Run:
|
|
```bash
|
|
./incusos-proxmox --status examples/lab-production.yaml
|
|
```
|
|
The status check includes automatic scrub_schedule healing.
|
|
|
|
|
|
## Section 3: Operations Center Setup
|
|
|
|
### Add OC Remote
|
|
|
|
```bash
|
|
operations-center remote add oc-lab https://192.168.102.110:8443 --auth-type tls
|
|
```
|
|
|
|
Accept the certificate fingerprint when prompted.
|
|
|
|
**Important**: The OC CLI does **not** support the `remote:` suffix syntax
|
|
that the Incus CLI uses. Instead, switch to the remote first, then run
|
|
commands without a remote suffix:
|
|
|
|
```bash
|
|
operations-center remote switch oc-lab
|
|
```
|
|
|
|
### Verify OC
|
|
|
|
```bash
|
|
operations-center admin os show
|
|
```
|
|
|
|
Actual output:
|
|
|
|
| PROPERTY | VALUE |
|
|
|---------------|--------------------------------------------------------|
|
|
| hostname | lab-oc |
|
|
| os_version | 202602210344 |
|
|
| kernel | 6.12.13 |
|
|
| architecture | x86_64 |
|
|
| uptime | 4390 |
|
|
| addresses | 192.168.102.110/22 (mgmt), fd42:...:1 (incusbr0) |
|
|
| storage_disks | /dev/sda (53.7GB, QEMU) |
|
|
| storage_pools | local (zfs, /dev/sda4) |
|
|
|
|
### Check Application Status
|
|
|
|
```bash
|
|
operations-center admin os application list
|
|
```
|
|
|
|
Actual output:
|
|
|
|
| NAME | STATUS |
|
|
|-------------------|---------|
|
|
| operations-center | running |
|
|
|
|
### Check for Updates
|
|
|
|
```bash
|
|
operations-center provisioning update list
|
|
```
|
|
|
|
Shows available IncusOS updates. Updates can be applied via the OC web UI
|
|
or CLI.
|
|
|
|
### Service Status
|
|
|
|
```bash
|
|
operations-center admin os service list
|
|
```
|
|
|
|
Actual output:
|
|
|
|
| NAME | ENABLED |
|
|
|----------|---------|
|
|
| ovn | false |
|
|
| syslog | false |
|
|
| fan | false |
|
|
| bgp | false |
|
|
| dns | false |
|
|
| metricsA | false |
|
|
| metricsB | false |
|
|
|
|
### Web UI Access
|
|
|
|
Open `https://192.168.102.110:8443/ui/` in your browser. You need the
|
|
PKCS#12 client certificate imported (see Section 1). The web UI provides
|
|
a dashboard view of the OC server. After adding Incus nodes to OC
|
|
(optional), the dashboard shows cluster health.
|
|
|
|
**Note**: OC deployed with a standard ISO acts as a monitoring dashboard.
|
|
For full OC node management (provisioning, cluster orchestration), nodes
|
|
must boot from an OC-provisioned ISO. See
|
|
[Operations Center Guide](operations-center-guide.md) for the full hybrid
|
|
workflow.
|
|
|
|
|
|
## Section 4: Cluster Formation
|
|
|
|
### 4.1 Set Specific IP Addresses
|
|
|
|
IncusOS nodes default to `core.https_address: :8443` (wildcard). Clustering
|
|
requires specific routable IPs so nodes can address each other.
|
|
|
|
```bash
|
|
incus config set lab-node-01: core.https_address 192.168.102.111:8443
|
|
incus config set lab-node-02: core.https_address 192.168.102.112:8443
|
|
incus config set lab-node-03: core.https_address 192.168.102.113:8443
|
|
```
|
|
|
|
Verify on each node:
|
|
|
|
```bash
|
|
incus config get lab-node-01: core.https_address
|
|
incus config get lab-node-02: core.https_address
|
|
incus config get lab-node-03: core.https_address
|
|
```
|
|
|
|
Each should return `IP:8443`.
|
|
|
|
### 4.2 Enable Clustering on Init Node
|
|
|
|
```bash
|
|
incus cluster enable lab-node-01: lab-node-01
|
|
```
|
|
|
|
**Note the syntax**: TWO arguments — `lab-node-01:` (remote with trailing
|
|
colon) and `lab-node-01` (member name). This is NOT `lab-node-01:lab-node-01`.
|
|
|
|
### 4.3 Fix Init Node Remote
|
|
|
|
Enabling clustering regenerates the TLS certificate. The new cert may only
|
|
have SANs for `127.0.0.1` and `::1`, breaking the remote.
|
|
|
|
```bash
|
|
incus remote switch local
|
|
incus remote remove lab-node-01
|
|
incus remote add lab-node-01 https://192.168.102.111:8443 --accept-certificate
|
|
```
|
|
|
|
Verify:
|
|
|
|
```bash
|
|
incus cluster list lab-node-01:
|
|
```
|
|
|
|
Expected output:
|
|
|
|
| NAME | URL | ROLES | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATE | MESSAGE |
|
|
|-------------|----------------------------------|--------------------------|--------------|----------------|-------------|--------|-------------------|
|
|
| lab-node-01 | https://192.168.102.111:8443 | database-leader, database | x86_64 | default | | ONLINE | Fully operational |
|
|
|
|
### 4.4 Join Node-02
|
|
|
|
Generate a join token on the init node:
|
|
|
|
```bash
|
|
incus cluster add lab-node-01:lab-node-02
|
|
```
|
|
|
|
This outputs a long base64 token. Use it immediately — tokens expire.
|
|
|
|
Join node-02 to the cluster (automated, non-interactive):
|
|
|
|
```bash
|
|
printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join lab-node-01: lab-node-02:
|
|
```
|
|
|
|
The five prompts answered by `printf`:
|
|
1. IP address → accept default (node's IP)
|
|
2. Member name → accept default (matches token)
|
|
3. "All existing data is lost" → `yes`
|
|
4. `source` for storage pool "local" → `local/incus`
|
|
5. `zfs.pool_name` for pool "local" → `local/incus`
|
|
|
|
No storage/network cleanup needed — `apply_defaults: false` means node-02
|
|
has no pre-existing Incus storage pool or network.
|
|
|
|
Fix the remote after join (new cluster cert):
|
|
|
|
```bash
|
|
incus remote remove lab-node-02
|
|
incus remote add lab-node-02 https://192.168.102.112:8443 --accept-certificate
|
|
```
|
|
|
|
### 4.5 Join Node-03
|
|
|
|
Same procedure:
|
|
|
|
```bash
|
|
incus cluster add lab-node-01:lab-node-03
|
|
printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join lab-node-01: lab-node-03:
|
|
```
|
|
|
|
Fix the remote:
|
|
|
|
```bash
|
|
incus remote remove lab-node-03
|
|
incus remote add lab-node-03 https://192.168.102.113:8443 --accept-certificate
|
|
```
|
|
|
|
### 4.6 Verify Cluster
|
|
|
|
```bash
|
|
incus cluster list lab-node-01:
|
|
```
|
|
|
|
Expected output — 3 nodes, all ONLINE:
|
|
|
|
| NAME | URL | ROLES | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATE | MESSAGE |
|
|
|-------------|----------------------------------|--------------------------|--------------|----------------|-------------|--------|-------------------|
|
|
| lab-node-01 | https://192.168.102.111:8443 | database-leader, database | x86_64 | default | | ONLINE | Fully operational |
|
|
| lab-node-02 | https://192.168.102.112:8443 | database | x86_64 | default | | ONLINE | Fully operational |
|
|
| lab-node-03 | https://192.168.102.113:8443 | database | x86_64 | default | | ONLINE | Fully operational |
|
|
|
|
Verify storage pool exists on all members:
|
|
|
|
```bash
|
|
incus storage show lab-node-01:local
|
|
incus storage show lab-node-01:local --target lab-node-02
|
|
incus storage show lab-node-01:local --target lab-node-03
|
|
```
|
|
|
|
Verify the default network:
|
|
|
|
```bash
|
|
incus network list lab-node-01:
|
|
```
|
|
|
|
|
|
## Section 5: Bridge Networking Baseline
|
|
|
|
Before setting up OVN, establish the baseline: bridge networks are
|
|
node-local. This demonstrates why OVN is needed.
|
|
|
|
### Same-Node Communication
|
|
|
|
Launch 2 containers on the same node. **Important**: use `--target` to force
|
|
placement — without it, the cluster scheduler may place containers on
|
|
different nodes automatically:
|
|
|
|
```bash
|
|
incus launch images:debian/12 lab-node-01:test-bridge-a --target lab-node-01
|
|
incus launch images:debian/12 lab-node-01:test-bridge-b --target lab-node-01
|
|
```
|
|
|
|
Wait for them to get IPs:
|
|
|
|
```bash
|
|
incus list lab-node-01: --columns ns4 --format csv | grep test-bridge
|
|
```
|
|
|
|
Ping between them:
|
|
|
|
```bash
|
|
IP_B=$(incus list lab-node-01:test-bridge-b --columns 4 --format csv | cut -d' ' -f1)
|
|
incus exec lab-node-01:test-bridge-a -- ping -c 3 "$IP_B"
|
|
```
|
|
|
|
Actual result: 0% packet loss, ~0.024ms latency. Same bridge, same node — works.
|
|
|
|
### Cross-Node Communication (Fails)
|
|
|
|
Launch a container on a different node:
|
|
|
|
```bash
|
|
incus launch images:debian/12 lab-node-01:test-bridge-c --target lab-node-02
|
|
```
|
|
|
|
Wait for IP:
|
|
|
|
```bash
|
|
incus list lab-node-01: --columns ns4 --format csv | grep test-bridge
|
|
```
|
|
|
|
Ping from node-01 to node-02:
|
|
|
|
```bash
|
|
IP_C=$(incus list lab-node-01:test-bridge-c --columns 4 --format csv | cut -d' ' -f1)
|
|
incus exec lab-node-01:test-bridge-a -- ping -c 3 -W 2 "$IP_C"
|
|
```
|
|
|
|
Actual result: **100% packet loss**. Bridge networks are node-local — there is
|
|
no L2 path between `incusbr0` on node-01 and `incusbr0` on node-02. Each
|
|
node's bridge has the same subnet (e.g., 10.251.22.1/24) but they are
|
|
separate L2 domains.
|
|
|
|
### Internet Access
|
|
|
|
NAT to the internet works from any node:
|
|
|
|
```bash
|
|
incus exec lab-node-01:test-bridge-a -- ping -c 3 1.1.1.1
|
|
```
|
|
|
|
Actual result: 0% packet loss, ~10ms latency. Each bridge provides NAT via
|
|
the host's management interface.
|
|
|
|
### Cleanup
|
|
|
|
```bash
|
|
incus delete lab-node-01:test-bridge-a --force
|
|
incus delete lab-node-01:test-bridge-b --force
|
|
incus delete lab-node-01:test-bridge-c --force
|
|
```
|
|
|
|
|
|
## Section 6: OVN Overlay Networking
|
|
|
|
OVN provides a cross-node L2 overlay using Geneve tunnels. After this
|
|
section, containers on any node can communicate transparently.
|
|
|
|
### 6.1 Deploy OVN Control Plane
|
|
|
|
Launch a Debian container on node-01 to host the OVN central services:
|
|
|
|
```bash
|
|
incus launch images:debian/12 lab-node-01:ovn-central --target lab-node-01
|
|
```
|
|
|
|
Install OVN:
|
|
|
|
```bash
|
|
incus exec lab-node-01:ovn-central -- apt-get update
|
|
incus exec lab-node-01:ovn-central -- apt-get install -y ovn-central ovn-host
|
|
```
|
|
|
|
Configure OVN to listen on all interfaces:
|
|
|
|
```bash
|
|
incus exec lab-node-01:ovn-central -- ovn-nbctl set-connection ptcp:6641:0.0.0.0
|
|
incus exec lab-node-01:ovn-central -- ovn-sbctl set-connection ptcp:6642:0.0.0.0
|
|
```
|
|
|
|
Add proxy devices to expose NB and SB ports on the host's LAN IP:
|
|
|
|
```bash
|
|
incus config device add lab-node-01:ovn-central nb-proxy proxy \
|
|
listen=tcp:192.168.102.111:6641 connect=tcp:127.0.0.1:6641
|
|
incus config device add lab-node-01:ovn-central sb-proxy proxy \
|
|
listen=tcp:192.168.102.111:6642 connect=tcp:127.0.0.1:6642
|
|
```
|
|
|
|
Verify the ports are reachable:
|
|
|
|
```bash
|
|
curl -s --connect-timeout 2 telnet://192.168.102.111:6641 || echo "NB port open"
|
|
curl -s --connect-timeout 2 telnet://192.168.102.111:6642 || echo "SB port open"
|
|
```
|
|
|
|
### 6.2 Enable OVN on All IncusOS Nodes
|
|
|
|
OVN services are disabled by default on IncusOS. Enable them on every node
|
|
via the IncusOS REST API. The `database` field points to the **southbound**
|
|
DB (port 6642, not 6641).
|
|
|
|
**Node-01:**
|
|
|
|
```bash
|
|
incus query lab-node-01:/os/1.0/services/ovn --request PUT --data '{
|
|
"config": {
|
|
"database": "tcp:192.168.102.111:6642",
|
|
"enabled": true,
|
|
"tunnel_address": "192.168.102.111",
|
|
"tunnel_protocol": "geneve"
|
|
},
|
|
"state": {}
|
|
}'
|
|
```
|
|
|
|
**Node-02:**
|
|
|
|
```bash
|
|
incus query lab-node-02:/os/1.0/services/ovn --request PUT --data '{
|
|
"config": {
|
|
"database": "tcp:192.168.102.111:6642",
|
|
"enabled": true,
|
|
"tunnel_address": "192.168.102.112",
|
|
"tunnel_protocol": "geneve"
|
|
},
|
|
"state": {}
|
|
}'
|
|
```
|
|
|
|
**Node-03:**
|
|
|
|
```bash
|
|
incus query lab-node-03:/os/1.0/services/ovn --request PUT --data '{
|
|
"config": {
|
|
"database": "tcp:192.168.102.111:6642",
|
|
"enabled": true,
|
|
"tunnel_address": "192.168.102.113",
|
|
"tunnel_protocol": "geneve"
|
|
},
|
|
"state": {}
|
|
}'
|
|
```
|
|
|
|
Each call should return `{}` on success.
|
|
|
|
### 6.3 Configure Incus OVN Connection
|
|
|
|
Point Incus to the **northbound** DB (port 6641):
|
|
|
|
```bash
|
|
incus config set lab-node-01: network.ovn.northbound_connection tcp:192.168.102.111:6641
|
|
```
|
|
|
|
### 6.4 Assign OVN Chassis Role
|
|
|
|
Every node that will host OVN workloads needs the `ovn-chassis` role:
|
|
|
|
```bash
|
|
incus cluster role add lab-node-01:lab-node-01 ovn-chassis
|
|
incus cluster role add lab-node-01:lab-node-02 ovn-chassis
|
|
incus cluster role add lab-node-01:lab-node-03 ovn-chassis
|
|
```
|
|
|
|
Verify:
|
|
|
|
```bash
|
|
incus cluster list lab-node-01:
|
|
```
|
|
|
|
The ROLES column should now include `ovn-chassis` for each member.
|
|
|
|
### 6.5 Create UPLINK Physical Network
|
|
|
|
The UPLINK network provides the bridge between OVN virtual networks and
|
|
the physical LAN. It uses the two-step cluster pattern: per-member
|
|
`--target` first, then cluster-wide create.
|
|
|
|
**Per-member configuration** (one per node):
|
|
|
|
**Important**: IncusOS names its management NIC `mgmt`, NOT `ens18`. Using
|
|
`parent=ens18` will fail with "Parent interface 'ens18' not found". Verify
|
|
with: `incus query lab-node-01:/os/1.0/system/network`
|
|
|
|
```bash
|
|
incus network create lab-node-01:UPLINK --type physical --target lab-node-01 \
|
|
parent=mgmt
|
|
|
|
incus network create lab-node-01:UPLINK --type physical --target lab-node-02 \
|
|
parent=mgmt
|
|
|
|
incus network create lab-node-01:UPLINK --type physical --target lab-node-03 \
|
|
parent=mgmt
|
|
```
|
|
|
|
**Cluster-wide create** with shared settings:
|
|
|
|
```bash
|
|
incus network create lab-node-01:UPLINK --type physical \
|
|
ipv4.ovn.ranges=192.168.103.200-192.168.103.210 \
|
|
ipv4.gateway=192.168.100.1/22 \
|
|
dns.nameservers=192.168.100.1
|
|
```
|
|
|
|
### 6.6 Create OVN Network (net-prod)
|
|
|
|
```bash
|
|
incus network create lab-node-01:net-prod --type=ovn network=UPLINK \
|
|
ipv4.address=10.10.10.1/24 \
|
|
ipv4.nat=true \
|
|
ipv6.address=none
|
|
```
|
|
|
|
Verify:
|
|
|
|
```bash
|
|
incus network list lab-node-01:
|
|
```
|
|
|
|
Should show both `incusbr0` (bridge, per-node) and `net-prod` (ovn, cluster-wide).
|
|
|
|
### 6.7 Verify Cross-Node OVN Connectivity
|
|
|
|
Launch containers on different nodes, attached to net-prod:
|
|
|
|
```bash
|
|
incus launch images:debian/12 lab-node-01:test-ovn-a --network net-prod --target lab-node-01
|
|
incus launch images:debian/12 lab-node-01:test-ovn-b --network net-prod --target lab-node-02
|
|
incus launch images:debian/12 lab-node-01:test-ovn-c --network net-prod --target lab-node-03
|
|
```
|
|
|
|
Wait for IPs and list:
|
|
|
|
```bash
|
|
incus list lab-node-01: --columns nst4 --format csv | grep test-ovn
|
|
```
|
|
|
|
Cross-node ping (node-01 → node-02):
|
|
|
|
```bash
|
|
IP_B=$(incus list lab-node-01:test-ovn-b --columns 4 --format csv | cut -d' ' -f1)
|
|
incus exec lab-node-01:test-ovn-a -- ping -c 3 "$IP_B"
|
|
```
|
|
|
|
Actual result: 0% packet loss, ~0.09-0.8ms latency. OVN provides transparent
|
|
L2 connectivity via Geneve tunnels.
|
|
|
|
Cross-node ping (node-01 → node-03):
|
|
|
|
```bash
|
|
IP_C=$(incus list lab-node-01:test-ovn-c --columns 4 --format csv | cut -d' ' -f1)
|
|
incus exec lab-node-01:test-ovn-a -- ping -c 3 "$IP_C"
|
|
```
|
|
|
|
Internet access through OVN:
|
|
|
|
```bash
|
|
incus exec lab-node-01:test-ovn-a -- ping -c 3 1.1.1.1
|
|
```
|
|
|
|
Clean up test containers:
|
|
|
|
```bash
|
|
incus delete lab-node-01:test-ovn-a --force
|
|
incus delete lab-node-01:test-ovn-b --force
|
|
incus delete lab-node-01:test-ovn-c --force
|
|
```
|
|
|
|
|
|
## Section 7: Mixed Workloads
|
|
|
|
Deploy a realistic workload mix: web servers, application containers, and
|
|
VMs configured for live migration.
|
|
|
|
### 7.1 Containers on net-prod
|
|
|
|
Deploy containers with targeted placement across nodes:
|
|
|
|
```bash
|
|
# Web servers
|
|
incus launch images:debian/12 lab-node-01:prod-web-01 --network net-prod --target lab-node-01
|
|
incus launch images:debian/12 lab-node-01:prod-web-02 --network net-prod --target lab-node-02
|
|
|
|
# Application container
|
|
incus launch images:debian/12 lab-node-01:prod-api-01 --network net-prod --target lab-node-03
|
|
```
|
|
|
|
Install nginx on the web servers:
|
|
|
|
```bash
|
|
incus exec lab-node-01:prod-web-01 -- bash -c "apt-get update && apt-get install -y nginx"
|
|
incus exec lab-node-01:prod-web-02 -- bash -c "apt-get update && apt-get install -y nginx"
|
|
```
|
|
|
|
Set distinct content to verify load balancing later:
|
|
|
|
```bash
|
|
incus exec lab-node-01:prod-web-01 -- bash -c "echo 'Server: prod-web-01' > /var/www/html/index.html"
|
|
incus exec lab-node-01:prod-web-02 -- bash -c "echo 'Server: prod-web-02' > /var/www/html/index.html"
|
|
```
|
|
|
|
Install nginx on the API container:
|
|
|
|
```bash
|
|
incus exec lab-node-01:prod-api-01 -- bash -c "apt-get update && apt-get install -y nginx"
|
|
incus exec lab-node-01:prod-api-01 -- bash -c "echo 'API: prod-api-01' > /var/www/html/index.html"
|
|
```
|
|
|
|
### 7.2 VMs (Migration-Ready)
|
|
|
|
Deploy VMs with live migration configuration:
|
|
|
|
```bash
|
|
incus launch images:debian/12 lab-node-01:prod-db-01 --vm --network net-prod --target lab-node-01
|
|
incus launch images:debian/12 lab-node-01:prod-app-01 --vm --network net-prod --target lab-node-02
|
|
```
|
|
|
|
VMs may take longer to boot than containers (~30-60s for image download +
|
|
boot). If the VMs show as STOPPED, start them explicitly:
|
|
|
|
```bash
|
|
incus start lab-node-01:prod-db-01
|
|
incus start lab-node-01:prod-app-01
|
|
```
|
|
|
|
Wait for the VM agent to become available, then verify:
|
|
|
|
```bash
|
|
# Check VM agent is running
|
|
incus exec lab-node-01:prod-db-01 -- uname -a
|
|
incus exec lab-node-01:prod-app-01 -- uname -a
|
|
```
|
|
|
|
### 7.3 Configure VMs for Live Migration
|
|
|
|
**Critical**: use `limits.cpu` as a **range** (e.g., `0-1`), not an integer.
|
|
Without the range, QEMU sets `maxcpus` based on the host's CPU count, which
|
|
varies across nodes and breaks migration with `Missing section footer for
|
|
ICH9LPC`.
|
|
|
|
Stop VMs before configuring `migration.stateful`:
|
|
|
|
```bash
|
|
incus stop lab-node-01:prod-db-01
|
|
incus stop lab-node-01:prod-app-01
|
|
```
|
|
|
|
Configure migration settings:
|
|
|
|
```bash
|
|
# prod-db-01
|
|
incus config set lab-node-01:prod-db-01 limits.cpu=0-1
|
|
incus config set lab-node-01:prod-db-01 migration.stateful=true
|
|
incus config device set lab-node-01:prod-db-01 root size.state=2GiB
|
|
|
|
# prod-app-01
|
|
incus config set lab-node-01:prod-app-01 limits.cpu=0-1
|
|
incus config set lab-node-01:prod-app-01 migration.stateful=true
|
|
incus config device set lab-node-01:prod-app-01 root size.state=2GiB
|
|
```
|
|
|
|
Start the VMs:
|
|
|
|
```bash
|
|
incus start lab-node-01:prod-db-01
|
|
incus start lab-node-01:prod-app-01
|
|
```
|
|
|
|
### 7.4 Workload Distribution
|
|
|
|
View the full workload distribution:
|
|
|
|
```bash
|
|
incus list lab-node-01: --columns nstL4 --format table
|
|
```
|
|
|
|
Expected layout:
|
|
|
|
| NAME | STATE | TYPE | LOCATION | IPV4 |
|
|
|-------------|---------|-----------------|-------------|-----------------------|
|
|
| ovn-central | RUNNING | CONTAINER | lab-node-01 | ... |
|
|
| prod-web-01 | RUNNING | CONTAINER | lab-node-01 | 10.10.10.x (net-prod) |
|
|
| prod-db-01 | RUNNING | VIRTUAL-MACHINE | lab-node-01 | 10.10.10.x (net-prod) |
|
|
| prod-web-02 | RUNNING | CONTAINER | lab-node-02 | 10.10.10.x (net-prod) |
|
|
| prod-app-01 | RUNNING | VIRTUAL-MACHINE | lab-node-02 | 10.10.10.x (net-prod) |
|
|
| prod-api-01 | RUNNING | CONTAINER | lab-node-03 | 10.10.10.x (net-prod) |
|
|
|
|
|
|
## Section 8: Network Isolation & Security
|
|
|
|
### 8.1 Create Isolated Network
|
|
|
|
```bash
|
|
incus network create lab-node-01:net-isolated --type=ovn network=UPLINK \
|
|
ipv4.address=10.10.20.1/24 \
|
|
ipv4.nat=true \
|
|
ipv6.address=none
|
|
```
|
|
|
|
### 8.2 Launch Isolated Containers
|
|
|
|
```bash
|
|
incus launch images:debian/12 lab-node-01:iso-app-01 --network net-isolated --target lab-node-01
|
|
incus launch images:debian/12 lab-node-01:iso-app-02 --network net-isolated --target lab-node-02
|
|
```
|
|
|
|
### 8.3 Verify Network Isolation
|
|
|
|
Containers on net-isolated can reach each other:
|
|
|
|
```bash
|
|
IP_ISO2=$(incus list lab-node-01:iso-app-02 --columns 4 --format csv | cut -d' ' -f1)
|
|
incus exec lab-node-01:iso-app-01 -- ping -c 3 "$IP_ISO2"
|
|
```
|
|
|
|
Actual result: 0% packet loss, ~0.15-0.5ms latency. Containers on the
|
|
same OVN network can reach each other across nodes.
|
|
|
|
But net-prod **cannot** reach net-isolated:
|
|
|
|
```bash
|
|
incus exec lab-node-01:prod-web-01 -- ping -c 3 -W 2 "$IP_ISO2"
|
|
```
|
|
|
|
Actual result: **100% packet loss**. Different OVN networks are fully
|
|
isolated — separate L2 domains, no routing between them.
|
|
|
|
### 8.4 Create Network ACL
|
|
|
|
Create an ACL that blocks ICMP from a specific source:
|
|
|
|
```bash
|
|
incus network acl create lab-node-01:block-ping
|
|
incus network acl rule add lab-node-01:block-ping ingress \
|
|
action=drop protocol=icmp4 \
|
|
source=10.10.10.0/24 \
|
|
description="Block ICMP from net-prod subnet"
|
|
```
|
|
|
|
### 8.5 Apply and Test ACL
|
|
|
|
Apply the ACL to net-isolated:
|
|
|
|
```bash
|
|
incus network set lab-node-01:net-isolated security.acls=block-ping
|
|
```
|
|
|
|
Verify ICMP is blocked between net-isolated containers (since they match
|
|
the source range — adjust the ACL source for targeted blocking):
|
|
|
|
```bash
|
|
incus exec lab-node-01:iso-app-01 -- ping -c 3 -W 2 "$IP_ISO2"
|
|
```
|
|
|
|
Remove the ACL:
|
|
|
|
```bash
|
|
incus network unset lab-node-01:net-isolated security.acls
|
|
```
|
|
|
|
Verify ICMP works again:
|
|
|
|
```bash
|
|
incus exec lab-node-01:iso-app-01 -- ping -c 3 "$IP_ISO2"
|
|
```
|
|
|
|
### 8.6 Network Peering
|
|
|
|
Connect net-prod and net-isolated so containers on both networks can
|
|
communicate. Peering is bilateral — create a peer on both sides:
|
|
|
|
```bash
|
|
# From net-prod's perspective
|
|
incus network peer create lab-node-01:net-prod peer-to-isolated net-isolated \
|
|
--description "Peer to isolated network"
|
|
|
|
# From net-isolated's perspective
|
|
incus network peer create lab-node-01:net-isolated peer-to-prod net-prod \
|
|
--description "Peer to production network"
|
|
```
|
|
|
|
### 8.7 Verify Peering
|
|
|
|
Cross-network ping (prod → isolated):
|
|
|
|
```bash
|
|
incus exec lab-node-01:prod-web-01 -- ping -c 3 "$IP_ISO2"
|
|
```
|
|
|
|
Actual result: 0% packet loss with TTL=62 (64 - 2 router hops),
|
|
confirming traffic traverses the OVN routers on both sides of the peering.
|
|
|
|
Cross-network ping (isolated → prod):
|
|
|
|
```bash
|
|
IP_WEB1=$(incus list lab-node-01:prod-web-01 --columns 4 --format csv | cut -d' ' -f1)
|
|
incus exec lab-node-01:iso-app-01 -- ping -c 3 "$IP_WEB1"
|
|
```
|
|
|
|
### 8.8 Remove Peering
|
|
|
|
```bash
|
|
incus network peer delete lab-node-01:net-prod peer-to-isolated
|
|
incus network peer delete lab-node-01:net-isolated peer-to-prod
|
|
```
|
|
|
|
Verify isolation is restored:
|
|
|
|
```bash
|
|
incus exec lab-node-01:prod-web-01 -- ping -c 3 -W 2 "$IP_ISO2"
|
|
```
|
|
|
|
Expected: 100% packet loss. Networks are isolated again.
|
|
|
|
Clean up isolated containers:
|
|
|
|
```bash
|
|
incus delete lab-node-01:iso-app-01 --force
|
|
incus delete lab-node-01:iso-app-02 --force
|
|
```
|
|
|
|
|
|
## Section 9: Load Balancers & Network Forwards
|
|
|
|
### 9.1 Create OVN Load Balancer
|
|
|
|
Create a load balancer with a VIP from the UPLINK range:
|
|
|
|
```bash
|
|
incus network load-balancer create lab-node-01:net-prod 192.168.103.200
|
|
```
|
|
|
|
Add backend servers. **Important**: backends require the instance's **IP
|
|
address**, not its name. Get the IPs first:
|
|
|
|
```bash
|
|
WEB1_IP=$(incus list lab-node-01:prod-web-01 --columns 4 --format csv | cut -d' ' -f1)
|
|
WEB2_IP=$(incus list lab-node-01:prod-web-02 --columns 4 --format csv | cut -d' ' -f1)
|
|
echo "prod-web-01: $WEB1_IP, prod-web-02: $WEB2_IP"
|
|
```
|
|
|
|
Add backends using IP addresses:
|
|
|
|
```bash
|
|
incus network load-balancer backend add lab-node-01:net-prod 192.168.103.200 \
|
|
web-01 "$WEB1_IP" 80
|
|
incus network load-balancer backend add lab-node-01:net-prod 192.168.103.200 \
|
|
web-02 "$WEB2_IP" 80
|
|
```
|
|
|
|
Add a port mapping:
|
|
|
|
```bash
|
|
incus network load-balancer port add lab-node-01:net-prod 192.168.103.200 \
|
|
tcp 80 web-01,web-02
|
|
```
|
|
|
|
### 9.2 Test Load Balancer
|
|
|
|
From your dev machine (must be on the same VLAN or have routing to
|
|
192.168.103.0/24):
|
|
|
|
```bash
|
|
for i in $(seq 1 6); do
|
|
curl -s http://192.168.103.200
|
|
done
|
|
```
|
|
|
|
Actual output:
|
|
|
|
```
|
|
Server: prod-web-01
|
|
Server: prod-web-01
|
|
Server: prod-web-01
|
|
Server: prod-web-02
|
|
Server: prod-web-02
|
|
Server: prod-web-02
|
|
```
|
|
|
|
OVN uses connection-based hashing (not round-robin). Multiple requests from
|
|
the same source will typically hit the same backend. Different source ports
|
|
or connections may hit different backends.
|
|
|
|
### 9.3 Create Network Forward
|
|
|
|
Network forwards expose internal services on LAN IPs. Forward
|
|
tcp:8080 → prod-api-01:80. **Like LB backends, forwards require IP addresses**:
|
|
|
|
```bash
|
|
API_IP=$(incus list lab-node-01:prod-api-01 --columns 4 --format csv | cut -d' ' -f1)
|
|
|
|
incus network forward create lab-node-01:net-prod 192.168.103.201
|
|
incus network forward port add lab-node-01:net-prod 192.168.103.201 \
|
|
tcp 8080 "$API_IP" 80
|
|
```
|
|
|
|
### 9.4 Test Network Forward
|
|
|
|
```bash
|
|
curl -s http://192.168.103.201:8080
|
|
```
|
|
|
|
Actual output: `API: prod-api-01`
|
|
|
|
### 9.5 DNS Resolution
|
|
|
|
OVN provides per-network DNS. Containers can resolve each other by
|
|
hostname:
|
|
|
|
```bash
|
|
incus exec lab-node-01:prod-web-01 -- bash -c "apt-get install -y dnsutils && dig +short prod-web-02.incus"
|
|
```
|
|
|
|
Actual output: `10.10.10.3` — OVN DNS resolves instance names within
|
|
each network.
|
|
|
|
|
|
## Section 10: Live Migration
|
|
|
|
### 10.1 Verify Migration Readiness
|
|
|
|
Check that VMs have the required configuration:
|
|
|
|
```bash
|
|
for vm in prod-db-01 prod-app-01; do
|
|
echo "=== $vm ==="
|
|
incus config get lab-node-01:$vm limits.cpu
|
|
incus config get lab-node-01:$vm migration.stateful
|
|
incus config device get lab-node-01:$vm root size.state
|
|
done
|
|
```
|
|
|
|
Expected: `0-1`, `true`, `2GiB` for each VM.
|
|
|
|
### 10.2 Create Heartbeat Service
|
|
|
|
Create a simple counter in prod-db-01 to verify state continuity across
|
|
migration:
|
|
|
|
```bash
|
|
incus exec lab-node-01:prod-db-01 -- bash -c '
|
|
mkdir -p /tmp/heartbeat
|
|
nohup bash -c "i=0; while true; do echo \$i > /tmp/heartbeat/counter; i=\$((i+1)); sleep 1; done" \
|
|
> /dev/null 2>&1 &
|
|
echo "Heartbeat started"
|
|
'
|
|
```
|
|
|
|
Read the counter:
|
|
|
|
```bash
|
|
incus exec lab-node-01:prod-db-01 -- cat /tmp/heartbeat/counter
|
|
```
|
|
|
|
Note the value. After migration, the counter should continue from where
|
|
it left off (live migration preserves running state).
|
|
|
|
### 10.3 Live Migration Round-Trip
|
|
|
|
Check current location:
|
|
|
|
```bash
|
|
incus list lab-node-01:prod-db-01 --columns nL --format csv
|
|
```
|
|
|
|
**Migrate node-01 → node-02:**
|
|
|
|
```bash
|
|
time incus move lab-node-01:prod-db-01 --target lab-node-02
|
|
```
|
|
|
|
Actual result: 7.347s (~140 MB/s). Wait for the VM agent to reconnect:
|
|
|
|
```bash
|
|
sleep 4
|
|
incus exec lab-node-01:prod-db-01 -- cat /tmp/heartbeat/counter
|
|
```
|
|
|
|
Counter went from 9 → 25. The heartbeat process was never interrupted —
|
|
it continued counting during migration.
|
|
|
|
**Migrate node-02 → node-03:**
|
|
|
|
```bash
|
|
time incus move lab-node-01:prod-db-01 --target lab-node-03
|
|
sleep 4
|
|
incus exec lab-node-01:prod-db-01 -- cat /tmp/heartbeat/counter
|
|
```
|
|
|
|
Actual result: 7.379s. Counter went to 41.
|
|
|
|
**Migrate node-03 → node-01 (back to origin):**
|
|
|
|
```bash
|
|
time incus move lab-node-01:prod-db-01 --target lab-node-01
|
|
sleep 4
|
|
incus exec lab-node-01:prod-db-01 -- cat /tmp/heartbeat/counter
|
|
```
|
|
|
|
Actual result: 6.896s. Counter went to 56.
|
|
|
|
Verify the VM is back on node-01:
|
|
|
|
```bash
|
|
incus list lab-node-01:prod-db-01 --columns nL --format csv
|
|
```
|
|
|
|
### 10.4 Active I/O During Migration
|
|
|
|
Start a continuous write inside the VM:
|
|
|
|
```bash
|
|
incus exec lab-node-01:prod-db-01 -- bash -c '
|
|
dd if=/dev/urandom of=/tmp/testfile bs=1M count=100 &
|
|
echo "Write started, PID: $!"
|
|
'
|
|
```
|
|
|
|
Migrate while I/O is active:
|
|
|
|
```bash
|
|
time incus move lab-node-01:prod-db-01 --target lab-node-02
|
|
sleep 4
|
|
```
|
|
|
|
Verify the file exists and is intact:
|
|
|
|
```bash
|
|
incus exec lab-node-01:prod-db-01 -- ls -la /tmp/testfile
|
|
incus exec lab-node-01:prod-db-01 -- md5sum /tmp/testfile
|
|
```
|
|
|
|
Move back:
|
|
|
|
```bash
|
|
incus move lab-node-01:prod-db-01 --target lab-node-01
|
|
sleep 4
|
|
```
|
|
|
|
### 10.5 Stateful Stop/Restore
|
|
|
|
Stateful stop saves VM memory to disk. On start, the VM resumes exactly
|
|
where it was:
|
|
|
|
```bash
|
|
# Note the heartbeat counter
|
|
incus exec lab-node-01:prod-app-01 -- bash -c '
|
|
mkdir -p /tmp/heartbeat
|
|
echo 42 > /tmp/heartbeat/counter
|
|
cat /tmp/heartbeat/counter
|
|
'
|
|
```
|
|
|
|
Stateful stop:
|
|
|
|
```bash
|
|
incus stop lab-node-01:prod-app-01 --stateful
|
|
```
|
|
|
|
Start (resumes from saved state):
|
|
|
|
```bash
|
|
incus start lab-node-01:prod-app-01
|
|
sleep 4
|
|
incus exec lab-node-01:prod-app-01 -- cat /tmp/heartbeat/counter
|
|
```
|
|
|
|
Expected: `42` — the file (and entire VM state) is preserved.
|
|
|
|
**If the restore fails** (e.g., from a `limits.cpu` mismatch), discard the
|
|
saved state:
|
|
|
|
```bash
|
|
incus start lab-node-01:prod-app-01 --stateless
|
|
```
|
|
|
|
|
|
## Section 11: Cluster Lifecycle
|
|
|
|
### 11.1 Evacuation & Restore
|
|
|
|
Evacuate node-02. All workloads are moved to other nodes:
|
|
|
|
```bash
|
|
incus cluster evacuate lab-node-01:lab-node-02 --force
|
|
```
|
|
|
|
Check workload distribution — nothing on node-02:
|
|
|
|
```bash
|
|
incus list lab-node-01: --columns nstL --format table
|
|
```
|
|
|
|
Actual behavior: VMs with `migration.stateful=true` are live-migrated
|
|
(prod-app-01 migrated to lab-node-03). Containers are stopped and moved
|
|
(prod-web-02 stopped, moved to lab-node-03, then started). The `--force`
|
|
flag skips confirmation prompts.
|
|
|
|
**Note**: if VMs lack the `limits.cpu` range fix, use `--action stop`
|
|
instead to avoid migration failures:
|
|
```bash
|
|
incus cluster evacuate lab-node-01:lab-node-02 --force --action stop
|
|
```
|
|
|
|
Verify node-02 shows EVACUATED:
|
|
|
|
```bash
|
|
incus cluster list lab-node-01:
|
|
```
|
|
|
|
Restore node-02 (workloads return):
|
|
|
|
```bash
|
|
incus cluster restore lab-node-01:lab-node-02 --force
|
|
```
|
|
|
|
Verify all workloads are back:
|
|
|
|
```bash
|
|
incus list lab-node-01: --columns nstL --format table
|
|
incus cluster list lab-node-01:
|
|
```
|
|
|
|
All nodes should show ONLINE.
|
|
|
|
### 11.2 Node Failure Simulation
|
|
|
|
A Proxmox hard-stop on a VM simulates a crash. The Incus cluster heartbeat
|
|
detects the failure in ~40 seconds. After the node is restarted:
|
|
|
|
1. The node auto-rejoins the cluster (~60s)
|
|
2. Containers auto-start
|
|
3. VMs that were running resume
|
|
|
|
**Procedure** (document only — do not execute while OVN is running unless
|
|
you can tolerate temporary network disruption):
|
|
|
|
```bash
|
|
# Simulate crash: hard-stop via Proxmox API
|
|
# curl -s -k -X POST "https://192.168.1.29:8006/api2/json/nodes/pve/qemu/912/status/stop" ...
|
|
|
|
# Wait for heartbeat detection (~40s)
|
|
# incus cluster list lab-node-01:
|
|
# → lab-node-02 shows OFFLINE
|
|
|
|
# Restart via Proxmox
|
|
# curl -s -k -X POST "https://192.168.1.29:8006/api2/json/nodes/pve/qemu/912/status/start" ...
|
|
|
|
# Wait for auto-rejoin (~60s)
|
|
# incus cluster list lab-node-01:
|
|
# → lab-node-02 shows ONLINE
|
|
```
|
|
|
|
### 11.3 Node Replacement
|
|
|
|
Full procedure: evacuate a node, remove it from the cluster, destroy the
|
|
VM, deploy a fresh node, and join it back. This tests the complete lifecycle.
|
|
|
|
**Step 1: Evacuate node-03:**
|
|
|
|
```bash
|
|
incus cluster evacuate lab-node-01:lab-node-03 --force --action stop
|
|
```
|
|
|
|
**Step 2: Remove from cluster:**
|
|
|
|
```bash
|
|
printf "yes\n" | incus cluster remove lab-node-01:lab-node-03 --force
|
|
```
|
|
|
|
**Note**: `incus cluster remove` prompts "Are you really sure?" even with
|
|
`--force`. The `printf` pipes `yes` for automation.
|
|
|
|
**Step 3: Clean up the remote:**
|
|
|
|
```bash
|
|
incus remote remove lab-node-03
|
|
```
|
|
|
|
**Step 4: Destroy and redeploy the VM.** Use `incusos-proxmox` to destroy
|
|
just node-03 and redeploy it. The simplest approach: create a single-VM
|
|
config or use the replacement config pattern:
|
|
|
|
```bash
|
|
# Destroy just node-03 via Proxmox API (VMID 913)
|
|
# Then redeploy with incusos-proxmox using a config that only defines node-03
|
|
```
|
|
|
|
Alternatively, if you have a `lab-replace.yaml` config for the replacement
|
|
node:
|
|
|
|
```bash
|
|
./incusos-proxmox --yes examples/lab-replace.yaml
|
|
```
|
|
|
|
**Step 5: Join the fresh node to the cluster:**
|
|
|
|
```bash
|
|
# Set specific IP
|
|
incus config set lab-node-03: core.https_address 192.168.102.113:8443
|
|
|
|
# Generate join token
|
|
incus cluster add lab-node-01:lab-node-03
|
|
|
|
# Join
|
|
printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join lab-node-01: lab-node-03:
|
|
|
|
# Fix remote
|
|
incus remote remove lab-node-03
|
|
incus remote add lab-node-03 https://192.168.102.113:8443 --accept-certificate
|
|
```
|
|
|
|
**Step 6: Re-enable OVN on the replacement node:**
|
|
|
|
```bash
|
|
incus query lab-node-03:/os/1.0/services/ovn --request PUT --data '{
|
|
"config": {
|
|
"database": "tcp:192.168.102.111:6642",
|
|
"enabled": true,
|
|
"tunnel_address": "192.168.102.113",
|
|
"tunnel_protocol": "geneve"
|
|
},
|
|
"state": {}
|
|
}'
|
|
|
|
incus cluster role add lab-node-01:lab-node-03 ovn-chassis
|
|
```
|
|
|
|
**Step 7: Verify:**
|
|
|
|
```bash
|
|
incus cluster list lab-node-01:
|
|
```
|
|
|
|
All 3 nodes should be ONLINE with `ovn-chassis` role.
|
|
|
|
### 11.4 Cluster Rebalancing
|
|
|
|
Enable automatic workload rebalancing. When a new node joins (or workloads
|
|
are unevenly distributed), Incus redistributes VMs:
|
|
|
|
```bash
|
|
incus config set lab-node-01: cluster.rebalance.interval=1
|
|
incus config set lab-node-01: cluster.rebalance.threshold=10
|
|
incus config set lab-node-01: cluster.rebalance.batch=2
|
|
incus config set lab-node-01: cluster.rebalance.cooldown=5m
|
|
```
|
|
|
|
**Important**: only VMs with `migration.stateful=true` are rebalanced.
|
|
Containers are NOT auto-rebalanced.
|
|
|
|
Monitor rebalancing:
|
|
|
|
```bash
|
|
incus list lab-node-01: --columns nstL --format table
|
|
```
|
|
|
|
Disable rebalancing when done testing:
|
|
|
|
```bash
|
|
incus config unset lab-node-01: cluster.rebalance.interval
|
|
incus config unset lab-node-01: cluster.rebalance.threshold
|
|
```
|
|
|
|
|
|
## Section 12: OC Dashboard
|
|
|
|
**Important**: Switch to the OC remote first. The OC CLI does not support
|
|
`remote:` suffix syntax:
|
|
|
|
```bash
|
|
operations-center remote switch oc-lab
|
|
```
|
|
|
|
### OC Server Information
|
|
|
|
```bash
|
|
operations-center admin os show
|
|
```
|
|
|
|
Actual output:
|
|
|
|
| PROPERTY | VALUE |
|
|
|---------------|--------------------------------------------------------|
|
|
| hostname | lab-oc |
|
|
| os_version | 202602210344 |
|
|
| kernel | 6.12.13 |
|
|
| architecture | x86_64 |
|
|
| uptime | 4390 |
|
|
| addresses | 192.168.102.110/22 (mgmt), fd42:...:1 (incusbr0) |
|
|
| storage_disks | /dev/sda (53.7GB, QEMU) |
|
|
| storage_pools | local (zfs, /dev/sda4) |
|
|
|
|
### Application Status
|
|
|
|
```bash
|
|
operations-center admin os application list
|
|
```
|
|
|
|
Actual output:
|
|
|
|
| NAME | STATUS |
|
|
|-------------------|---------|
|
|
| operations-center | running |
|
|
|
|
### Service Status
|
|
|
|
```bash
|
|
operations-center admin os service list
|
|
```
|
|
|
|
Actual output:
|
|
|
|
| NAME | ENABLED |
|
|
|----------|---------|
|
|
| ovn | false |
|
|
| syslog | false |
|
|
| fan | false |
|
|
| bgp | false |
|
|
| dns | false |
|
|
| metricsA | false |
|
|
| metricsB | false |
|
|
|
|
### Web UI
|
|
|
|
The OC web UI at `https://192.168.102.110:8443/ui/` provides:
|
|
|
|
- **Dashboard**: server overview with resource utilization
|
|
- **Updates**: available IncusOS updates
|
|
- **Provisioning**: token management (for OC-provisioned deployments)
|
|
- **System**: OC configuration and certificates
|
|
|
|
**Limitation**: OC deployed with a standard ISO cannot manage the Incus
|
|
cluster nodes. The nodes are independent — they were deployed with a
|
|
standard IncusOS ISO, not an OC-provisioned one. For full OC node
|
|
management (cluster orchestration, application deployment, monitoring),
|
|
nodes must boot from an OC-provisioned ISO.
|
|
|
|
See [Operations Center Guide](operations-center-guide.md) for the full
|
|
hybrid deployment workflow with OC-provisioned ISOs.
|
|
|
|
|
|
## Section 13: Cleanup
|
|
|
|
### Delete All Workloads
|
|
|
|
```bash
|
|
# Delete containers
|
|
for c in prod-web-01 prod-web-02 prod-api-01; do
|
|
incus delete lab-node-01:$c --force
|
|
done
|
|
|
|
# Delete VMs
|
|
for vm in prod-db-01 prod-app-01; do
|
|
incus delete lab-node-01:$vm --force
|
|
done
|
|
```
|
|
|
|
### Remove OVN Networks
|
|
|
|
```bash
|
|
# Delete OVN networks
|
|
incus network delete lab-node-01:net-prod
|
|
incus network delete lab-node-01:net-isolated 2>/dev/null || true
|
|
|
|
# Delete UPLINK
|
|
incus network delete lab-node-01:UPLINK
|
|
```
|
|
|
|
### Remove OVN Control Plane
|
|
|
|
```bash
|
|
incus delete lab-node-01:ovn-central --force
|
|
```
|
|
|
|
### Disable OVN Services
|
|
|
|
```bash
|
|
for node in lab-node-01 lab-node-02 lab-node-03; do
|
|
incus query "$node":/os/1.0/services/ovn --request PUT --data '{
|
|
"config": {
|
|
"enabled": false
|
|
},
|
|
"state": {}
|
|
}'
|
|
done
|
|
```
|
|
|
|
### Infrastructure Options
|
|
|
|
**Keep infrastructure** (stop VMs, keep on disk for later):
|
|
|
|
```bash
|
|
./incusos-proxmox --lab-down examples/lab-production.yaml
|
|
```
|
|
|
|
Restart later with:
|
|
|
|
```bash
|
|
./incusos-proxmox --lab-up examples/lab-production.yaml
|
|
```
|
|
|
|
**Full teardown** (destroy all VMs, remove ISOs, remotes, cache):
|
|
|
|
```bash
|
|
./incusos-proxmox --cleanup --deep examples/lab-production.yaml
|
|
```
|
|
|
|
|
|
## Section 14: Verification Checklist
|
|
|
|
| # | Check | Command | Expected |
|
|
|---|-------|---------|----------|
|
|
| 1 | All VMs running | `incusos-proxmox --status examples/lab-production.yaml` | 4 VMs running, port 8443 open |
|
|
| 2 | Scrub schedule healthy | `incus query lab-node-01:/os/1.0/system/storage` | `scrub_schedule: "0 4 * * 0"` |
|
|
| 3 | OC accessible | `operations-center remote switch oc-lab && operations-center admin os show` | Shows version, uptime |
|
|
| 4 | Cluster formed | `incus cluster list lab-node-01:` | 3 nodes ONLINE |
|
|
| 5 | Storage pool | `incus storage list lab-node-01:` | `local` pool on all members |
|
|
| 6 | Bridge isolation | Ping cross-node on incusbr0 | 100% loss (expected) |
|
|
| 7 | OVN connectivity | Ping cross-node on net-prod | 0% loss |
|
|
| 8 | Internet via OVN | `ping 1.1.1.1` from OVN container | 0% loss |
|
|
| 9 | Network isolation | Ping net-prod → net-isolated | 100% loss (expected) |
|
|
| 10 | Network peering | Peer + ping cross-network | 0% loss, TTL=62 |
|
|
| 11 | Load balancer | `curl http://192.168.103.200` | Backend response |
|
|
| 12 | Network forward | `curl http://192.168.103.201:8080` | API response |
|
|
| 13 | DNS resolution | `dig prod-web-02.incus` from container | Resolves to 10.10.10.x |
|
|
| 14 | VM live migration | `incus move` VM between nodes | State preserved |
|
|
| 15 | Cluster evacuation | `incus cluster evacuate` + `restore` | Workloads moved and returned |
|
|
| 16 | Stateful stop/start | `incus stop --stateful` + `start` | VM state preserved |
|
|
|
|
|
|
## Section 15: Quick Reference
|
|
|
|
### Cluster Command Syntax
|
|
|
|
| Command | Arguments | Notes |
|
|
|---------|-----------|-------|
|
|
| `incus cluster enable` | `remote: member-name` | TWO args (space between) |
|
|
| `incus cluster add` | `remote:member-name` | ONE arg (no space) |
|
|
| `incus cluster join` | `init-remote: joining-remote:` | TWO args (space between) |
|
|
| `incus cluster remove` | `remote:member-name --force` | ONE arg; prompts even with `--force` |
|
|
| `incus cluster evacuate` | `remote:member-name` | ONE arg (no space) |
|
|
| `incus cluster restore` | `remote:member-name` | ONE arg (no space) |
|
|
| `incus config set` | `remote: key value` | Remote with trailing colon + space |
|
|
| `incus storage show` | `remote:pool` | ONE arg (no space) |
|
|
| `incus storage show` | `remote:pool --target member` | `--target` for member-specific |
|
|
|
|
### OVN Setup Cheat Sheet
|
|
|
|
```bash
|
|
# 1. Deploy OVN container
|
|
incus launch images:debian/12 REMOTE:ovn-central --target NODE
|
|
incus exec REMOTE:ovn-central -- apt-get install -y ovn-central ovn-host
|
|
incus exec REMOTE:ovn-central -- ovn-nbctl set-connection ptcp:6641:0.0.0.0
|
|
incus exec REMOTE:ovn-central -- ovn-sbctl set-connection ptcp:6642:0.0.0.0
|
|
incus config device add REMOTE:ovn-central nb-proxy proxy listen=tcp:HOST_IP:6641 connect=tcp:127.0.0.1:6641
|
|
incus config device add REMOTE:ovn-central sb-proxy proxy listen=tcp:HOST_IP:6642 connect=tcp:127.0.0.1:6642
|
|
|
|
# 2. Enable OVN on each IncusOS node
|
|
incus query NODE:/os/1.0/services/ovn --request PUT --data '{"config":{"database":"tcp:HOST_IP:6642","enabled":true,"tunnel_address":"NODE_IP","tunnel_protocol":"geneve"},"state":{}}'
|
|
|
|
# 3. Configure Incus
|
|
incus config set REMOTE: network.ovn.northbound_connection tcp:HOST_IP:6641
|
|
incus cluster role add REMOTE:MEMBER ovn-chassis # for each member
|
|
|
|
# 4. Create UPLINK (per-member then cluster-wide)
|
|
incus network create REMOTE:UPLINK --type physical --target MEMBER parent=mgmt # each member
|
|
incus network create REMOTE:UPLINK --type physical ipv4.ovn.ranges=RANGE ipv4.gateway=GW/PREFIX
|
|
|
|
# 5. Create OVN network
|
|
incus network create REMOTE:net-name --type=ovn network=UPLINK ipv4.address=SUBNET ipv4.nat=true
|
|
```
|
|
|
|
### Migration Readiness Checklist
|
|
|
|
| Setting | Value | Why |
|
|
|---------|-------|-----|
|
|
| `limits.cpu` | Range (e.g., `0-1`) | Fixed QEMU topology across hosts |
|
|
| `migration.stateful` | `true` | Enables live migration |
|
|
| `root size.state` | `2GiB` (or `4GiB` for 3-4 vCPUs) | Space for memory state file |
|
|
|
|
Configure while VM is **stopped**:
|
|
|
|
```bash
|
|
incus stop REMOTE:VM
|
|
incus config set REMOTE:VM limits.cpu=0-1
|
|
incus config set REMOTE:VM migration.stateful=true
|
|
incus config device set REMOTE:VM root size.state=2GiB
|
|
incus start REMOTE:VM
|
|
```
|
|
|
|
### Troubleshooting
|
|
|
|
| Symptom | Cause | Fix |
|
|
|---------|-------|-----|
|
|
| Port 8443 not reachable after boot | Boot still in progress or crontab bug | Wait 180s; check scrub_schedule via API |
|
|
| `scrub_schedule` empty | Crontab race condition | `incusos-proxmox --status` auto-heals |
|
|
| `Missing section footer for ICH9LPC` on migration | `limits.cpu` set as integer | Set as range: `limits.cpu=0-1` |
|
|
| `VM agent isn't currently running` after migration | Agent reconnecting | `sleep 4` after migration |
|
|
| `db.sock not found` on OVN config | OVN service not enabled on IncusOS | Enable via `/os/1.0/services/ovn` API |
|
|
| Cross-node ping fails (bridge) | Bridge networks are node-local | Use OVN network instead |
|
|
| `zfs load-key: Raw key too short` | TPM corruption from premature VM stop | Destroy and redeploy VM |
|
|
| Cluster join fails with "pool already exists" | `apply_defaults: true` on joining node | Use `apply_defaults: false` or run 8-command cleanup |
|
|
| OC cannot manage cluster nodes | Nodes deployed with standard ISO | Use OC-provisioned ISO for full integration |
|
|
| `CPUID vnmi` warning during migration | Cosmetic QEMU check | Safe to ignore |
|
|
| "Parent interface 'ens18' not found" | IncusOS names its NIC `mgmt` | Use `parent=mgmt` for UPLINK network |
|
|
| "Invalid target address" on LB backend | Backend needs IP, not instance name | Use instance IP address (e.g., `10.10.10.2`) |
|
|
| OC CLI "Invalid number of arguments" | OC CLI doesn't support `remote:` suffix | Use `operations-center remote switch NAME` first |
|
|
| Container placed on wrong node | Cluster auto-schedules without `--target` | Use `--target NODE` for explicit placement |
|