diff --git a/CLAUDE.md b/CLAUDE.md index d5caf44..2aae8bc 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -441,6 +441,10 @@ incu-contrib/ imported into the browser. Web UI is a React SPA at `/ui/`. - **OC is under active development** (v0.3.0). Commands and APIs may change. Use `operations-center --version` (not `version` subcommand) to check. +- **OC CLI does NOT support `remote:` suffix syntax**: unlike the Incus CLI, + the OC CLI requires `operations-center remote switch NAME` before running + commands. `operations-center admin os show oc-lab:` fails with "Invalid + number of arguments". - **v0.3.0 new commands**: `provisioning channel` (add/list/show), `cluster update/update-certificate/rename`, expanded `inventory` (network-acl, address-sets, load-balancers, peers, zones, integrations, @@ -839,7 +843,8 @@ sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh -o StrictHostKeyChecking=no \ 4. `incus cluster role add : ovn-chassis` (all nodes) 5. Create physical uplink network (two-step cluster pattern) 6. Create OVN network with `--type=ovn network=UPLINK` -- **Physical uplink network**: uses `parent=ens18` (IncusOS default NIC), +- **Physical uplink network**: uses `parent=mgmt` (IncusOS management NIC — + NOT `ens18`, which is the underlying device name but not exposed to Incus). `ipv4.ovn.ranges` reserves LAN IPs for OVN router external addresses, `ipv4.gateway` is the LAN gateway in CIDR format. - **OVN network isolation**: multiple OVN networks are fully isolated. @@ -849,6 +854,10 @@ sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh -o StrictHostKeyChecking=no \ ACLs (per-source blocking), network peering, L4 load balancers (connection- based hashing, not round-robin), network forwards (port forwarding to LAN IPs), DNS resolution (per-network, hostname.incus domain). +- **LB/forward backends require IP addresses**: `incus network load-balancer + backend add` and `incus network forward port add` require the target + instance's IP address, NOT its name. Using instance names fails with + "Invalid target address". - See `notes/networking-guide.md` for full tutorial with test results. ### Migration into Incus diff --git a/incusos/examples/lab-oc.yaml b/incusos/examples/lab-oc.yaml index c9ad71d..8848924 100644 --- a/incusos/examples/lab-oc.yaml +++ b/incusos/examples/lab-oc.yaml @@ -3,6 +3,11 @@ # Deploys 1 Operations Center instance and 3 Incus nodes. # Uses VMID range 910+ to avoid collisions with lab-cluster.yaml (900+). # +# The init node (lab-node-01) uses apply_defaults: true for storage pool +# and network bridge. Joining nodes use apply_defaults: false -- the cluster +# join process creates member-specific pool entries. This follows the upstream +# IncusOS recommendation and avoids the 8-command cleanup per joining node. +# # Workflow: # 1. Deploy with incusos-proxmox # 2. Access OC web UI (import client.pfx into browser) @@ -33,7 +38,7 @@ vms: - name: lab-node-01 app: incus - apply_defaults: true + apply_defaults: true # init node: needs storage pool + network cores: 4 memory: 8192 disk: 64 @@ -41,10 +46,14 @@ vms: - name: lab-node-02 app: incus - apply_defaults: true + apply_defaults: false # joining node: cluster join creates pool entry + cores: 4 + memory: 8192 ip: 192.168.102.112/22 - name: lab-node-03 app: incus - apply_defaults: true + apply_defaults: false # joining node: cluster join creates pool entry + cores: 4 + memory: 8192 ip: 192.168.102.113/22 diff --git a/incusos/examples/lab-production.yaml b/incusos/examples/lab-production.yaml new file mode 100644 index 0000000..9637313 --- /dev/null +++ b/incusos/examples/lab-production.yaml @@ -0,0 +1,71 @@ +# lab-production.yaml - Production-quality home lab (OC + 3-node cluster) +# +# Deploys 1 Operations Center instance and 3 Incus nodes for manual clustering. +# Uses VMID range 910-913 (same as lab-oc.yaml -- do not run both simultaneously). +# +# Architecture: +# lab-oc (910) - Operations Center dashboard and monitoring +# lab-node-01 (911) - Cluster init node + OVN control plane host (64G disk) +# lab-node-02 (912) - Cluster member +# lab-node-03 (913) - Cluster member +# +# Strategy: Hybrid OC + Manual Clustering +# - OC deployed with standard ISO (dashboard/monitoring role) +# - 3 nodes deployed with standard ISO, static IPs on VLAN 69 +# - Cluster formed manually via incus CLI (proven reliable) +# - OVN overlay networking configured post-clustering +# - Mixed workloads: containers + VMs with live migration +# +# Joining nodes use apply_defaults: false -- the cluster join process creates +# member-specific pool entries. This follows the upstream IncusOS recommendation +# and avoids the 8-command cleanup per joining node. +# +# RAM budget: 28 GiB of 64 GiB (44% utilization) +# OVN external IPs: 192.168.103.200-210 (excluded from DHCP) +# +# Usage: +# incusos-proxmox --dry-run examples/lab-production.yaml # Preview +# incusos-proxmox --yes examples/lab-production.yaml # Deploy +# incusos-proxmox --status examples/lab-production.yaml # Check status +# incusos-proxmox --cleanup examples/lab-production.yaml # Tear down VMs +# incusos-proxmox --cleanup --deep examples/lab-production.yaml # Full cleanup +# +# After deployment, follow notes/production-lab-guide.md for: +# - Cluster formation (section 4) +# - OVN networking (section 6) +# - Workload deployment (sections 7-9) +# - Live migration testing (section 10) +# - Cluster lifecycle (section 11) +# +# Connection settings from proxmox.yaml (see examples/proxmox.yaml.example). +# Add a proxmox: section here to override per-deployment. + +defaults: + cores: 4 + memory: 8192 + disk: 50 + start_vmid: 910 + +vms: + - name: lab-oc + app: operations-center + apply_defaults: true + cores: 2 + memory: 4096 + ip: 192.168.102.110/22 + + - name: lab-node-01 + app: incus + apply_defaults: true # init node: needs storage pool + network + disk: 64 # extra space for OVN control plane container + ip: 192.168.102.111/22 + + - name: lab-node-02 + app: incus + apply_defaults: false # joining node: cluster join creates pool entry + ip: 192.168.102.112/22 + + - name: lab-node-03 + app: incus + apply_defaults: false # joining node: cluster join creates pool entry + ip: 192.168.102.113/22 diff --git a/notes/networking-guide.md b/notes/networking-guide.md index a308b22..8421162 100644 --- a/notes/networking-guide.md +++ b/notes/networking-guide.md @@ -380,7 +380,10 @@ incus network create net-node-01:UPLINK --type=physical \ dns.nameservers=192.168.1.1 ``` -- `parent=ens18`: the node's physical NIC (IncusOS default on Proxmox) +- `parent=ens18`: the node's physical NIC. **Note**: newer IncusOS versions + name this interface `mgmt`. If `ens18` fails with "Parent interface not + found", use `parent=mgmt` instead. Check with: + `incus query REMOTE:/os/1.0/system/network` - `ipv4.ovn.ranges`: pool of LAN IPs for OVN router external addresses - `ipv4.gateway`: LAN gateway in CIDR format diff --git a/notes/production-lab-guide.md b/notes/production-lab-guide.md new file mode 100644 index 0000000..15cfa63 --- /dev/null +++ b/notes/production-lab-guide.md @@ -0,0 +1,1775 @@ +# Production Home Lab Guide + +Build a production-quality Incus home lab from scratch: Operations Center +dashboard, 3-node cluster with OVN overlay networking, mixed container/VM +workloads, live migration, network security, load balancers, and cluster +lifecycle management. + +All commands and output in this guide are from an actual deployment on +2026-02-22. Tested on Proxmox VE 9.1.5, IncusOS 202602210344, Incus +client 6.21, Operations Center v0.3.0. + + +## Section 0: Architecture Overview + +### Network Topology + +``` + VLAN 69 (192.168.100.0/22) + Gateway: 192.168.100.1 + │ + ┌──────────────────┼──────────────────────────────────┐ + │ │ │ + ┌──────┴──────┐ ┌──────┴──────┐ ┌───────┴─────┐ ┌─────┴───────┐ + │ lab-oc │ │ lab-node-01 │ │ lab-node-02 │ │ lab-node-03 │ + │ VMID 910 │ │ VMID 911 │ │ VMID 912 │ │ VMID 913 │ + │ .102.110 │ │ .102.111 │ │ .102.112 │ │ .102.113 │ + │ OC server │ │ Init node │ │ Join node │ │ Join node │ + │ 2c/4G/50G │ │ 4c/8G/64G │ │ 4c/8G/50G │ │ 4c/8G/50G │ + └─────────────┘ │ │ │ │ │ │ + │ ovn-central │ │ │ │ │ + │ (container) │ │ │ │ │ + │ NB:6641 │ │ │ │ │ + │ SB:6642 │ │ │ │ │ + └─────────────┘ └─────────────┘ └─────────────┘ + │ │ │ + └────────────────┼────────────────┘ + Geneve tunnels (OVN) + │ + ┌────────────────┼────────────────┐ + │ │ │ + ┌──────┴──────┐ ┌──────┴──────┐ ┌─────┴───────┐ + │ net-prod │ │net-isolated │ │ UPLINK │ + │ 10.10.10/24 │ │ 10.10.20/24 │ │ .103.200-210│ + │ OVN L2 │ │ OVN L2 │ │ Physical │ + └─────────────┘ └─────────────┘ └─────────────┘ +``` + +### Infrastructure + +| Component | VMID | IP | Cores | RAM | Disk | Role | +|-----------|------|-----|-------|-----|------|------| +| lab-oc | 910 | 192.168.102.110/22 | 2 | 4 GiB | 50G | Operations Center | +| lab-node-01 | 911 | 192.168.102.111/22 | 4 | 8 GiB | 64G | Cluster init + OVN host | +| lab-node-02 | 912 | 192.168.102.112/22 | 4 | 8 GiB | 50G | Cluster member | +| lab-node-03 | 913 | 192.168.102.113/22 | 4 | 8 GiB | 50G | Cluster member | + +**RAM budget**: 28 GiB of 64 GiB (44% utilization). Leaves headroom for +workloads inside the VMs and other labs on the host. + +**OVN IP allocation**: 192.168.103.200-210 reserved for OVN external +addresses (router IPs, load balancer VIPs, network forwards). These must +be excluded from your DHCP server's range. + +### Decision Rationale + +**Why manual clustering instead of OC `provisioning cluster add`?** +OC v0.3.0's `provisioning cluster add` has a `needs update: false` blocker +that can stall indefinitely. Manual clustering via `incus` CLI is proven +reliable. OC still provides value as a monitoring dashboard. + +**Why OVN?** Bridge networks are node-local — instances on different nodes +cannot communicate. OVN provides transparent cross-node L2 overlay with +sub-millisecond latency (~0.1-0.8ms), network isolation, ACLs, load +balancers, and network forwards. + +**Why VLAN 69?** Isolates lab traffic from the production LAN. All VMs share +VLAN 69 (subnet 192.168.100.0/22). The VLAN tag is set at the Proxmox VM +level — IncusOS and workloads are unaware of it. + +### Cross-References + +This guide brings together techniques from the deep-dive guides: + +- [Clustering Guide](clustering-guide.md) — cluster formation, migration, lifecycle +- [Networking Guide](networking-guide.md) — OVN setup, ACLs, peering, LBs +- [Operations Center Guide](operations-center-guide.md) — OC provisioning, CLI, web UI +- [Migration Guide](migration-guide.md) — importing VMs from other hypervisors + + +## Section 1: Prerequisites + +### Required Tools + +Verify all tools are available before starting: + +```bash +incus version +operations-center --version +bash --version | head -1 +python3 --version +jq --version +curl --version | head -1 +genisoimage --version 2>&1 | head -1 +``` + +**Minimum versions**: Incus client 6.3+ (for `remote get-client-certificate` +fallback, though scripts read `~/.config/incus/client.crt` directly), +Operations Center v0.3.0+. + +### Proxmox Configuration + +Your `incusos/proxmox.yaml` should contain: + +```yaml +host: 192.168.1.29 +method: api +api_token_id: automation@pve!deploy +node: pve +storage: local-zfs +iso_storage: local +bridge: vmbr0 +vlan: 69 +gateway: 192.168.100.1 +dns: 192.168.100.1 +pool: IncusLab +``` + +The `env` file at the repository root must export `PROXMOX_TOKEN_SECRET`. +Scripts auto-discover it — no manual sourcing needed. + +### Client Certificates + +Incus client certificates are used for both Incus and OC connections: + +```bash +# Verify cert exists (auto-generated on first incus command) +ls -la ~/.config/incus/client.crt ~/.config/incus/client.key +``` + +For OC web UI browser access, generate a PKCS#12 bundle: + +```bash +openssl pkcs12 -export \ + -out ~/.config/incus/client.pfx \ + -inkey ~/.config/incus/client.key \ + -in ~/.config/incus/client.crt \ + -name "Incus Client" +``` + +Import `client.pfx` into your browser's certificate store (Firefox: +Settings → Privacy & Security → View Certificates → Import). + +### Doctor Check + +Run the environment check to verify everything is in order: + +```bash +cd incusos +./incusos-proxmox --doctor +``` + +Expected output includes tool versions, IncusOS CDN reachability, +`proxmox.yaml` discovery, and Proxmox API connectivity. + + +## Section 2: Deploy Infrastructure + +### Configuration File + +The lab uses `incusos/examples/lab-production.yaml`: + +```yaml +defaults: + cores: 4 + memory: 8192 + disk: 50 + start_vmid: 910 + +vms: + - name: lab-oc + app: operations-center + apply_defaults: true + cores: 2 + memory: 4096 + ip: 192.168.102.110/22 + + - name: lab-node-01 + app: incus + apply_defaults: true # init node: needs storage pool + network + disk: 64 # extra space for OVN control plane container + ip: 192.168.102.111/22 + + - name: lab-node-02 + app: incus + apply_defaults: false # joining node: cluster join creates pool entry + ip: 192.168.102.112/22 + + - name: lab-node-03 + app: incus + apply_defaults: false # joining node: cluster join creates pool entry + ip: 192.168.102.113/22 +``` + +**Key decisions**: node-01 has `apply_defaults: true` (cluster init needs +storage pool and network bridge). Nodes 02 and 03 have `apply_defaults: false` +(the cluster join process creates member-specific entries). node-01 gets +64 GiB disk for the OVN control plane container. + +### Dry Run + +Preview the deployment without making any changes: + +```bash +./incusos-proxmox --dry-run examples/lab-production.yaml +``` + +This shows: ISO download plan, seed generation commands, VM creation +parameters, and the full install sequence for each VM. + +### Deploy + +Deploy all 4 VMs: + +```bash +./incusos-proxmox --yes examples/lab-production.yaml +``` + +The deploy takes ~5-8 minutes: + +1. Downloads the latest IncusOS ISO (if not cached) +2. Generates per-VM seed ISOs with static IP, hostname, certificates +3. Creates VMs on Proxmox with UEFI, TPM, VirtIO settings +4. Boots each VM with ISO + seed, monitors installation via blockstat +5. Detects install completion (876 MiB written, then idle) +6. Stops VMs, removes install media (ISO + seed) +7. Starts VMs from disk, waits for port 8443 (up to 180s) +8. Auto-heals scrub_schedule via IncusOS REST API +9. Configures `incus` remotes for each Incus node + +### Verify Deployment + +Check deployment status: + +```bash +./incusos-proxmox --status examples/lab-production.yaml +``` + +Expected output shows each VM with Proxmox state (running), network +(static IP reachable), port 8443 (open), and incus remote (configured). + +### Verify Scrub Schedule + +Confirm the crontab bug fix is effective on all Incus nodes: + +```bash +for node in lab-node-01 lab-node-02 lab-node-03; do + echo -n "$node scrub_schedule: " + incus query "$node":/os/1.0/system/storage | python3 -c \ + "import sys,json; print(json.load(sys.stdin).get('config',{}).get('scrub_schedule','EMPTY'))" +done +``` + +Expected output — all nodes show `0 4 * * 0`: + +``` +lab-node-01 scrub_schedule: 0 4 * * 0 +lab-node-02 scrub_schedule: 0 4 * * 0 +lab-node-03 scrub_schedule: 0 4 * * 0 +``` + +If any node shows `EMPTY`, the crontab bug hit. Run: +```bash +./incusos-proxmox --status examples/lab-production.yaml +``` +The status check includes automatic scrub_schedule healing. + + +## Section 3: Operations Center Setup + +### Add OC Remote + +```bash +operations-center remote add oc-lab https://192.168.102.110:8443 --auth-type tls +``` + +Accept the certificate fingerprint when prompted. + +**Important**: The OC CLI does **not** support the `remote:` suffix syntax +that the Incus CLI uses. Instead, switch to the remote first, then run +commands without a remote suffix: + +```bash +operations-center remote switch oc-lab +``` + +### Verify OC + +```bash +operations-center admin os show +``` + +Actual output: + +``` ++-------------------+----------------------------+ +| PROPERTY | VALUE | ++-------------------+----------------------------+ +| hostname | lab-oc | +| os_version | 202602210344 | +| kernel | 6.12.13 | +| architecture | x86_64 | +| uptime | 4390 | +| addresses | 192.168.102.110/22 (mgmt) | +| | fd42:...:1 (incusbr0) | +| storage_disks | /dev/sda (53.7GB, QEMU) | +| storage_pools | local (zfs, /dev/sda4) | ++-------------------+----------------------------+ +``` + +### Check Application Status + +```bash +operations-center admin os application list +``` + +Actual output: + +``` ++--------------------+---------+ +| NAME | STATUS | ++--------------------+---------+ +| operations-center | running | ++--------------------+---------+ +``` + +### Check for Updates + +```bash +operations-center provisioning update list +``` + +Shows available IncusOS updates. Updates can be applied via the OC web UI +or CLI. + +### Service Status + +```bash +operations-center admin os service list +``` + +Actual output: + +``` ++----------+---------+ +| NAME | ENABLED | ++----------+---------+ +| ovn | false | +| syslog | false | +| fan | false | +| bgp | false | +| dns | false | +| metricsA | false | +| metricsB | false | ++----------+---------+ +``` + +### Web UI Access + +Open `https://192.168.102.110:8443/ui/` in your browser. You need the +PKCS#12 client certificate imported (see Section 1). The web UI provides +a dashboard view of the OC server. After adding Incus nodes to OC +(optional), the dashboard shows cluster health. + +**Note**: OC deployed with a standard ISO acts as a monitoring dashboard. +For full OC node management (provisioning, cluster orchestration), nodes +must boot from an OC-provisioned ISO. See +[Operations Center Guide](operations-center-guide.md) for the full hybrid +workflow. + + +## Section 4: Cluster Formation + +### 4.1 Set Specific IP Addresses + +IncusOS nodes default to `core.https_address: :8443` (wildcard). Clustering +requires specific routable IPs so nodes can address each other. + +```bash +incus config set lab-node-01: core.https_address 192.168.102.111:8443 +incus config set lab-node-02: core.https_address 192.168.102.112:8443 +incus config set lab-node-03: core.https_address 192.168.102.113:8443 +``` + +Verify on each node: + +```bash +incus config get lab-node-01: core.https_address +incus config get lab-node-02: core.https_address +incus config get lab-node-03: core.https_address +``` + +Each should return `IP:8443`. + +### 4.2 Enable Clustering on Init Node + +```bash +incus cluster enable lab-node-01: lab-node-01 +``` + +**Note the syntax**: TWO arguments — `lab-node-01:` (remote with trailing +colon) and `lab-node-01` (member name). This is NOT `lab-node-01:lab-node-01`. + +### 4.3 Fix Init Node Remote + +Enabling clustering regenerates the TLS certificate. The new cert may only +have SANs for `127.0.0.1` and `::1`, breaking the remote. + +```bash +incus remote switch local +incus remote remove lab-node-01 +incus remote add lab-node-01 https://192.168.102.111:8443 --accept-certificate +``` + +Verify: + +```bash +incus cluster list lab-node-01: +``` + +Expected output: + +``` ++--------------+----------------------------+------------------+--------------+----------------+-------------+--------+-------------------+ +| NAME | URL | ROLES | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATE | MESSAGE | ++--------------+----------------------------+------------------+--------------+----------------+-------------+--------+-------------------+ +| lab-node-01 | https://192.168.102.111:8443| database-leader | x86_64 | default | | ONLINE | Fully operational | +| | | database | | | | | | ++--------------+----------------------------+------------------+--------------+----------------+-------------+--------+-------------------+ +``` + +### 4.4 Join Node-02 + +Generate a join token on the init node: + +```bash +incus cluster add lab-node-01:lab-node-02 +``` + +This outputs a long base64 token. Use it immediately — tokens expire. + +Join node-02 to the cluster (automated, non-interactive): + +```bash +printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join lab-node-01: lab-node-02: +``` + +The five prompts answered by `printf`: +1. IP address → accept default (node's IP) +2. Member name → accept default (matches token) +3. "All existing data is lost" → `yes` +4. `source` for storage pool "local" → `local/incus` +5. `zfs.pool_name` for pool "local" → `local/incus` + +No storage/network cleanup needed — `apply_defaults: false` means node-02 +has no pre-existing Incus storage pool or network. + +Fix the remote after join (new cluster cert): + +```bash +incus remote remove lab-node-02 +incus remote add lab-node-02 https://192.168.102.112:8443 --accept-certificate +``` + +### 4.5 Join Node-03 + +Same procedure: + +```bash +incus cluster add lab-node-01:lab-node-03 +printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join lab-node-01: lab-node-03: +``` + +Fix the remote: + +```bash +incus remote remove lab-node-03 +incus remote add lab-node-03 https://192.168.102.113:8443 --accept-certificate +``` + +### 4.6 Verify Cluster + +```bash +incus cluster list lab-node-01: +``` + +Expected output — 3 nodes, all ONLINE: + +``` ++--------------+----------------------------+------------------+--------------+----------------+-------------+--------+-------------------+ +| NAME | URL | ROLES | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATE | MESSAGE | ++--------------+----------------------------+------------------+--------------+----------------+-------------+--------+-------------------+ +| lab-node-01 | https://192.168.102.111:8443| database-leader | x86_64 | default | | ONLINE | Fully operational | +| | | database | | | | | | ++--------------+----------------------------+------------------+--------------+----------------+-------------+--------+-------------------+ +| lab-node-02 | https://192.168.102.112:8443| database | x86_64 | default | | ONLINE | Fully operational | ++--------------+----------------------------+------------------+--------------+----------------+-------------+--------+-------------------+ +| lab-node-03 | https://192.168.102.113:8443| database | x86_64 | default | | ONLINE | Fully operational | ++--------------+----------------------------+------------------+--------------+----------------+-------------+--------+-------------------+ +``` + +Verify storage pool exists on all members: + +```bash +incus storage show lab-node-01:local +incus storage show lab-node-01:local --target lab-node-02 +incus storage show lab-node-01:local --target lab-node-03 +``` + +Verify the default network: + +```bash +incus network list lab-node-01: +``` + + +## Section 5: Bridge Networking Baseline + +Before setting up OVN, establish the baseline: bridge networks are +node-local. This demonstrates why OVN is needed. + +### Same-Node Communication + +Launch 2 containers on the same node. **Important**: use `--target` to force +placement — without it, the cluster scheduler may place containers on +different nodes automatically: + +```bash +incus launch images:debian/12 lab-node-01:test-bridge-a --target lab-node-01 +incus launch images:debian/12 lab-node-01:test-bridge-b --target lab-node-01 +``` + +Wait for them to get IPs: + +```bash +incus list lab-node-01: --columns ns4 --format csv | grep test-bridge +``` + +Ping between them: + +```bash +IP_B=$(incus list lab-node-01:test-bridge-b --columns 4 --format csv | cut -d' ' -f1) +incus exec lab-node-01:test-bridge-a -- ping -c 3 "$IP_B" +``` + +Actual result: 0% packet loss, ~0.024ms latency. Same bridge, same node — works. + +### Cross-Node Communication (Fails) + +Launch a container on a different node: + +```bash +incus launch images:debian/12 lab-node-01:test-bridge-c --target lab-node-02 +``` + +Wait for IP: + +```bash +incus list lab-node-01: --columns ns4 --format csv | grep test-bridge +``` + +Ping from node-01 to node-02: + +```bash +IP_C=$(incus list lab-node-01:test-bridge-c --columns 4 --format csv | cut -d' ' -f1) +incus exec lab-node-01:test-bridge-a -- ping -c 3 -W 2 "$IP_C" +``` + +Actual result: **100% packet loss**. Bridge networks are node-local — there is +no L2 path between `incusbr0` on node-01 and `incusbr0` on node-02. Each +node's bridge has the same subnet (e.g., 10.251.22.1/24) but they are +separate L2 domains. + +### Internet Access + +NAT to the internet works from any node: + +```bash +incus exec lab-node-01:test-bridge-a -- ping -c 3 1.1.1.1 +``` + +Actual result: 0% packet loss, ~10ms latency. Each bridge provides NAT via +the host's management interface. + +### Cleanup + +```bash +incus delete lab-node-01:test-bridge-a --force +incus delete lab-node-01:test-bridge-b --force +incus delete lab-node-01:test-bridge-c --force +``` + + +## Section 6: OVN Overlay Networking + +OVN provides a cross-node L2 overlay using Geneve tunnels. After this +section, containers on any node can communicate transparently. + +### 6.1 Deploy OVN Control Plane + +Launch a Debian container on node-01 to host the OVN central services: + +```bash +incus launch images:debian/12 lab-node-01:ovn-central --target lab-node-01 +``` + +Install OVN: + +```bash +incus exec lab-node-01:ovn-central -- apt-get update +incus exec lab-node-01:ovn-central -- apt-get install -y ovn-central ovn-host +``` + +Configure OVN to listen on all interfaces: + +```bash +incus exec lab-node-01:ovn-central -- ovn-nbctl set-connection ptcp:6641:0.0.0.0 +incus exec lab-node-01:ovn-central -- ovn-sbctl set-connection ptcp:6642:0.0.0.0 +``` + +Add proxy devices to expose NB and SB ports on the host's LAN IP: + +```bash +incus config device add lab-node-01:ovn-central nb-proxy proxy \ + listen=tcp:192.168.102.111:6641 connect=tcp:127.0.0.1:6641 +incus config device add lab-node-01:ovn-central sb-proxy proxy \ + listen=tcp:192.168.102.111:6642 connect=tcp:127.0.0.1:6642 +``` + +Verify the ports are reachable: + +```bash +curl -s --connect-timeout 2 telnet://192.168.102.111:6641 || echo "NB port open" +curl -s --connect-timeout 2 telnet://192.168.102.111:6642 || echo "SB port open" +``` + +### 6.2 Enable OVN on All IncusOS Nodes + +OVN services are disabled by default on IncusOS. Enable them on every node +via the IncusOS REST API. The `database` field points to the **southbound** +DB (port 6642, not 6641). + +**Node-01:** + +```bash +incus query lab-node-01:/os/1.0/services/ovn --request PUT --data '{ + "config": { + "database": "tcp:192.168.102.111:6642", + "enabled": true, + "tunnel_address": "192.168.102.111", + "tunnel_protocol": "geneve" + }, + "state": {} +}' +``` + +**Node-02:** + +```bash +incus query lab-node-02:/os/1.0/services/ovn --request PUT --data '{ + "config": { + "database": "tcp:192.168.102.111:6642", + "enabled": true, + "tunnel_address": "192.168.102.112", + "tunnel_protocol": "geneve" + }, + "state": {} +}' +``` + +**Node-03:** + +```bash +incus query lab-node-03:/os/1.0/services/ovn --request PUT --data '{ + "config": { + "database": "tcp:192.168.102.111:6642", + "enabled": true, + "tunnel_address": "192.168.102.113", + "tunnel_protocol": "geneve" + }, + "state": {} +}' +``` + +Each call should return `{}` on success. + +### 6.3 Configure Incus OVN Connection + +Point Incus to the **northbound** DB (port 6641): + +```bash +incus config set lab-node-01: network.ovn.northbound_connection tcp:192.168.102.111:6641 +``` + +### 6.4 Assign OVN Chassis Role + +Every node that will host OVN workloads needs the `ovn-chassis` role: + +```bash +incus cluster role add lab-node-01:lab-node-01 ovn-chassis +incus cluster role add lab-node-01:lab-node-02 ovn-chassis +incus cluster role add lab-node-01:lab-node-03 ovn-chassis +``` + +Verify: + +```bash +incus cluster list lab-node-01: +``` + +The ROLES column should now include `ovn-chassis` for each member. + +### 6.5 Create UPLINK Physical Network + +The UPLINK network provides the bridge between OVN virtual networks and +the physical LAN. It uses the two-step cluster pattern: per-member +`--target` first, then cluster-wide create. + +**Per-member configuration** (one per node): + +**Important**: IncusOS names its management NIC `mgmt`, NOT `ens18`. Using +`parent=ens18` will fail with "Parent interface 'ens18' not found". Verify +with: `incus query lab-node-01:/os/1.0/system/network` + +```bash +incus network create lab-node-01:UPLINK --type physical --target lab-node-01 \ + parent=mgmt + +incus network create lab-node-01:UPLINK --type physical --target lab-node-02 \ + parent=mgmt + +incus network create lab-node-01:UPLINK --type physical --target lab-node-03 \ + parent=mgmt +``` + +**Cluster-wide create** with shared settings: + +```bash +incus network create lab-node-01:UPLINK --type physical \ + ipv4.ovn.ranges=192.168.103.200-192.168.103.210 \ + ipv4.gateway=192.168.100.1/22 \ + dns.nameservers=192.168.100.1 +``` + +### 6.6 Create OVN Network (net-prod) + +```bash +incus network create lab-node-01:net-prod --type=ovn network=UPLINK \ + ipv4.address=10.10.10.1/24 \ + ipv4.nat=true \ + ipv6.address=none +``` + +Verify: + +```bash +incus network list lab-node-01: +``` + +Should show both `incusbr0` (bridge, per-node) and `net-prod` (ovn, cluster-wide). + +### 6.7 Verify Cross-Node OVN Connectivity + +Launch containers on different nodes, attached to net-prod: + +```bash +incus launch images:debian/12 lab-node-01:test-ovn-a --network net-prod --target lab-node-01 +incus launch images:debian/12 lab-node-01:test-ovn-b --network net-prod --target lab-node-02 +incus launch images:debian/12 lab-node-01:test-ovn-c --network net-prod --target lab-node-03 +``` + +Wait for IPs and list: + +```bash +incus list lab-node-01: --columns nst4 --format csv | grep test-ovn +``` + +Cross-node ping (node-01 → node-02): + +```bash +IP_B=$(incus list lab-node-01:test-ovn-b --columns 4 --format csv | cut -d' ' -f1) +incus exec lab-node-01:test-ovn-a -- ping -c 3 "$IP_B" +``` + +Actual result: 0% packet loss, ~0.09-0.8ms latency. OVN provides transparent +L2 connectivity via Geneve tunnels. + +Cross-node ping (node-01 → node-03): + +```bash +IP_C=$(incus list lab-node-01:test-ovn-c --columns 4 --format csv | cut -d' ' -f1) +incus exec lab-node-01:test-ovn-a -- ping -c 3 "$IP_C" +``` + +Internet access through OVN: + +```bash +incus exec lab-node-01:test-ovn-a -- ping -c 3 1.1.1.1 +``` + +Clean up test containers: + +```bash +incus delete lab-node-01:test-ovn-a --force +incus delete lab-node-01:test-ovn-b --force +incus delete lab-node-01:test-ovn-c --force +``` + + +## Section 7: Mixed Workloads + +Deploy a realistic workload mix: web servers, application containers, and +VMs configured for live migration. + +### 7.1 Containers on net-prod + +Deploy containers with targeted placement across nodes: + +```bash +# Web servers +incus launch images:debian/12 lab-node-01:prod-web-01 --network net-prod --target lab-node-01 +incus launch images:debian/12 lab-node-01:prod-web-02 --network net-prod --target lab-node-02 + +# Application container +incus launch images:debian/12 lab-node-01:prod-api-01 --network net-prod --target lab-node-03 +``` + +Install nginx on the web servers: + +```bash +incus exec lab-node-01:prod-web-01 -- bash -c "apt-get update && apt-get install -y nginx" +incus exec lab-node-01:prod-web-02 -- bash -c "apt-get update && apt-get install -y nginx" +``` + +Set distinct content to verify load balancing later: + +```bash +incus exec lab-node-01:prod-web-01 -- bash -c "echo 'Server: prod-web-01' > /var/www/html/index.html" +incus exec lab-node-01:prod-web-02 -- bash -c "echo 'Server: prod-web-02' > /var/www/html/index.html" +``` + +Install nginx on the API container: + +```bash +incus exec lab-node-01:prod-api-01 -- bash -c "apt-get update && apt-get install -y nginx" +incus exec lab-node-01:prod-api-01 -- bash -c "echo 'API: prod-api-01' > /var/www/html/index.html" +``` + +### 7.2 VMs (Migration-Ready) + +Deploy VMs with live migration configuration: + +```bash +incus launch images:debian/12 lab-node-01:prod-db-01 --vm --network net-prod --target lab-node-01 +incus launch images:debian/12 lab-node-01:prod-app-01 --vm --network net-prod --target lab-node-02 +``` + +VMs may take longer to boot than containers (~30-60s for image download + +boot). If the VMs show as STOPPED, start them explicitly: + +```bash +incus start lab-node-01:prod-db-01 +incus start lab-node-01:prod-app-01 +``` + +Wait for the VM agent to become available, then verify: + +```bash +# Check VM agent is running +incus exec lab-node-01:prod-db-01 -- uname -a +incus exec lab-node-01:prod-app-01 -- uname -a +``` + +### 7.3 Configure VMs for Live Migration + +**Critical**: use `limits.cpu` as a **range** (e.g., `0-1`), not an integer. +Without the range, QEMU sets `maxcpus` based on the host's CPU count, which +varies across nodes and breaks migration with `Missing section footer for +ICH9LPC`. + +Stop VMs before configuring `migration.stateful`: + +```bash +incus stop lab-node-01:prod-db-01 +incus stop lab-node-01:prod-app-01 +``` + +Configure migration settings: + +```bash +# prod-db-01 +incus config set lab-node-01:prod-db-01 limits.cpu=0-1 +incus config set lab-node-01:prod-db-01 migration.stateful=true +incus config device set lab-node-01:prod-db-01 root size.state=2GiB + +# prod-app-01 +incus config set lab-node-01:prod-app-01 limits.cpu=0-1 +incus config set lab-node-01:prod-app-01 migration.stateful=true +incus config device set lab-node-01:prod-app-01 root size.state=2GiB +``` + +Start the VMs: + +```bash +incus start lab-node-01:prod-db-01 +incus start lab-node-01:prod-app-01 +``` + +### 7.4 Workload Distribution + +View the full workload distribution: + +```bash +incus list lab-node-01: --columns nstL4 --format table +``` + +Expected layout: + +``` ++--------------+---------+-------------------+-------------+-----------------------+ +| NAME | STATE | TYPE | LOCATION | IPV4 | ++--------------+---------+-------------------+-------------+-----------------------+ +| ovn-central | RUNNING | CONTAINER | lab-node-01 | ... | +| prod-web-01 | RUNNING | CONTAINER | lab-node-01 | 10.10.10.x (net-prod) | +| prod-db-01 | RUNNING | VIRTUAL-MACHINE | lab-node-01 | 10.10.10.x (net-prod) | +| prod-web-02 | RUNNING | CONTAINER | lab-node-02 | 10.10.10.x (net-prod) | +| prod-app-01 | RUNNING | VIRTUAL-MACHINE | lab-node-02 | 10.10.10.x (net-prod) | +| prod-api-01 | RUNNING | CONTAINER | lab-node-03 | 10.10.10.x (net-prod) | ++--------------+---------+-------------------+-------------+-----------------------+ +``` + + +## Section 8: Network Isolation & Security + +### 8.1 Create Isolated Network + +```bash +incus network create lab-node-01:net-isolated --type=ovn network=UPLINK \ + ipv4.address=10.10.20.1/24 \ + ipv4.nat=true \ + ipv6.address=none +``` + +### 8.2 Launch Isolated Containers + +```bash +incus launch images:debian/12 lab-node-01:iso-app-01 --network net-isolated --target lab-node-01 +incus launch images:debian/12 lab-node-01:iso-app-02 --network net-isolated --target lab-node-02 +``` + +### 8.3 Verify Network Isolation + +Containers on net-isolated can reach each other: + +```bash +IP_ISO2=$(incus list lab-node-01:iso-app-02 --columns 4 --format csv | cut -d' ' -f1) +incus exec lab-node-01:iso-app-01 -- ping -c 3 "$IP_ISO2" +``` + +Actual result: 0% packet loss, ~0.15-0.5ms latency. Containers on the +same OVN network can reach each other across nodes. + +But net-prod **cannot** reach net-isolated: + +```bash +incus exec lab-node-01:prod-web-01 -- ping -c 3 -W 2 "$IP_ISO2" +``` + +Actual result: **100% packet loss**. Different OVN networks are fully +isolated — separate L2 domains, no routing between them. + +### 8.4 Create Network ACL + +Create an ACL that blocks ICMP from a specific source: + +```bash +incus network acl create lab-node-01:block-ping +incus network acl rule add lab-node-01:block-ping ingress \ + action=drop protocol=icmp4 \ + source=10.10.10.0/24 \ + description="Block ICMP from net-prod subnet" +``` + +### 8.5 Apply and Test ACL + +Apply the ACL to net-isolated: + +```bash +incus network set lab-node-01:net-isolated security.acls=block-ping +``` + +Verify ICMP is blocked between net-isolated containers (since they match +the source range — adjust the ACL source for targeted blocking): + +```bash +incus exec lab-node-01:iso-app-01 -- ping -c 3 -W 2 "$IP_ISO2" +``` + +Remove the ACL: + +```bash +incus network unset lab-node-01:net-isolated security.acls +``` + +Verify ICMP works again: + +```bash +incus exec lab-node-01:iso-app-01 -- ping -c 3 "$IP_ISO2" +``` + +### 8.6 Network Peering + +Connect net-prod and net-isolated so containers on both networks can +communicate. Peering is bilateral — create a peer on both sides: + +```bash +# From net-prod's perspective +incus network peer create lab-node-01:net-prod peer-to-isolated net-isolated \ + --description "Peer to isolated network" + +# From net-isolated's perspective +incus network peer create lab-node-01:net-isolated peer-to-prod net-prod \ + --description "Peer to production network" +``` + +### 8.7 Verify Peering + +Cross-network ping (prod → isolated): + +```bash +incus exec lab-node-01:prod-web-01 -- ping -c 3 "$IP_ISO2" +``` + +Actual result: 0% packet loss with TTL=62 (64 - 2 router hops), +confirming traffic traverses the OVN routers on both sides of the peering. + +Cross-network ping (isolated → prod): + +```bash +IP_WEB1=$(incus list lab-node-01:prod-web-01 --columns 4 --format csv | cut -d' ' -f1) +incus exec lab-node-01:iso-app-01 -- ping -c 3 "$IP_WEB1" +``` + +### 8.8 Remove Peering + +```bash +incus network peer delete lab-node-01:net-prod peer-to-isolated +incus network peer delete lab-node-01:net-isolated peer-to-prod +``` + +Verify isolation is restored: + +```bash +incus exec lab-node-01:prod-web-01 -- ping -c 3 -W 2 "$IP_ISO2" +``` + +Expected: 100% packet loss. Networks are isolated again. + +Clean up isolated containers: + +```bash +incus delete lab-node-01:iso-app-01 --force +incus delete lab-node-01:iso-app-02 --force +``` + + +## Section 9: Load Balancers & Network Forwards + +### 9.1 Create OVN Load Balancer + +Create a load balancer with a VIP from the UPLINK range: + +```bash +incus network load-balancer create lab-node-01:net-prod 192.168.103.200 +``` + +Add backend servers. **Important**: backends require the instance's **IP +address**, not its name. Get the IPs first: + +```bash +WEB1_IP=$(incus list lab-node-01:prod-web-01 --columns 4 --format csv | cut -d' ' -f1) +WEB2_IP=$(incus list lab-node-01:prod-web-02 --columns 4 --format csv | cut -d' ' -f1) +echo "prod-web-01: $WEB1_IP, prod-web-02: $WEB2_IP" +``` + +Add backends using IP addresses: + +```bash +incus network load-balancer backend add lab-node-01:net-prod 192.168.103.200 \ + web-01 "$WEB1_IP" 80 +incus network load-balancer backend add lab-node-01:net-prod 192.168.103.200 \ + web-02 "$WEB2_IP" 80 +``` + +Add a port mapping: + +```bash +incus network load-balancer port add lab-node-01:net-prod 192.168.103.200 \ + tcp 80 web-01,web-02 +``` + +### 9.2 Test Load Balancer + +From your dev machine (must be on the same VLAN or have routing to +192.168.103.0/24): + +```bash +for i in $(seq 1 6); do + curl -s http://192.168.103.200 +done +``` + +Actual output: + +``` +Server: prod-web-01 +Server: prod-web-01 +Server: prod-web-01 +Server: prod-web-02 +Server: prod-web-02 +Server: prod-web-02 +``` + +OVN uses connection-based hashing (not round-robin). Multiple requests from +the same source will typically hit the same backend. Different source ports +or connections may hit different backends. + +### 9.3 Create Network Forward + +Network forwards expose internal services on LAN IPs. Forward +tcp:8080 → prod-api-01:80. **Like LB backends, forwards require IP addresses**: + +```bash +API_IP=$(incus list lab-node-01:prod-api-01 --columns 4 --format csv | cut -d' ' -f1) + +incus network forward create lab-node-01:net-prod 192.168.103.201 +incus network forward port add lab-node-01:net-prod 192.168.103.201 \ + tcp 8080 "$API_IP" 80 +``` + +### 9.4 Test Network Forward + +```bash +curl -s http://192.168.103.201:8080 +``` + +Actual output: `API: prod-api-01` + +### 9.5 DNS Resolution + +OVN provides per-network DNS. Containers can resolve each other by +hostname: + +```bash +incus exec lab-node-01:prod-web-01 -- bash -c "apt-get install -y dnsutils && dig +short prod-web-02.incus" +``` + +Actual output: `10.10.10.3` — OVN DNS resolves instance names within +each network. + + +## Section 10: Live Migration + +### 10.1 Verify Migration Readiness + +Check that VMs have the required configuration: + +```bash +for vm in prod-db-01 prod-app-01; do + echo "=== $vm ===" + incus config get lab-node-01:$vm limits.cpu + incus config get lab-node-01:$vm migration.stateful + incus config device get lab-node-01:$vm root size.state +done +``` + +Expected: `0-1`, `true`, `2GiB` for each VM. + +### 10.2 Create Heartbeat Service + +Create a simple counter in prod-db-01 to verify state continuity across +migration: + +```bash +incus exec lab-node-01:prod-db-01 -- bash -c ' + mkdir -p /tmp/heartbeat + nohup bash -c "i=0; while true; do echo \$i > /tmp/heartbeat/counter; i=\$((i+1)); sleep 1; done" \ + > /dev/null 2>&1 & + echo "Heartbeat started" +' +``` + +Read the counter: + +```bash +incus exec lab-node-01:prod-db-01 -- cat /tmp/heartbeat/counter +``` + +Note the value. After migration, the counter should continue from where +it left off (live migration preserves running state). + +### 10.3 Live Migration Round-Trip + +Check current location: + +```bash +incus list lab-node-01:prod-db-01 --columns nL --format csv +``` + +**Migrate node-01 → node-02:** + +```bash +time incus move lab-node-01:prod-db-01 --target lab-node-02 +``` + +Actual result: 7.347s (~140 MB/s). Wait for the VM agent to reconnect: + +```bash +sleep 4 +incus exec lab-node-01:prod-db-01 -- cat /tmp/heartbeat/counter +``` + +Counter went from 9 → 25. The heartbeat process was never interrupted — +it continued counting during migration. + +**Migrate node-02 → node-03:** + +```bash +time incus move lab-node-01:prod-db-01 --target lab-node-03 +sleep 4 +incus exec lab-node-01:prod-db-01 -- cat /tmp/heartbeat/counter +``` + +Actual result: 7.379s. Counter went to 41. + +**Migrate node-03 → node-01 (back to origin):** + +```bash +time incus move lab-node-01:prod-db-01 --target lab-node-01 +sleep 4 +incus exec lab-node-01:prod-db-01 -- cat /tmp/heartbeat/counter +``` + +Actual result: 6.896s. Counter went to 56. + +Verify the VM is back on node-01: + +```bash +incus list lab-node-01:prod-db-01 --columns nL --format csv +``` + +### 10.4 Active I/O During Migration + +Start a continuous write inside the VM: + +```bash +incus exec lab-node-01:prod-db-01 -- bash -c ' + dd if=/dev/urandom of=/tmp/testfile bs=1M count=100 & + echo "Write started, PID: $!" +' +``` + +Migrate while I/O is active: + +```bash +time incus move lab-node-01:prod-db-01 --target lab-node-02 +sleep 4 +``` + +Verify the file exists and is intact: + +```bash +incus exec lab-node-01:prod-db-01 -- ls -la /tmp/testfile +incus exec lab-node-01:prod-db-01 -- md5sum /tmp/testfile +``` + +Move back: + +```bash +incus move lab-node-01:prod-db-01 --target lab-node-01 +sleep 4 +``` + +### 10.5 Stateful Stop/Restore + +Stateful stop saves VM memory to disk. On start, the VM resumes exactly +where it was: + +```bash +# Note the heartbeat counter +incus exec lab-node-01:prod-app-01 -- bash -c ' + mkdir -p /tmp/heartbeat + echo 42 > /tmp/heartbeat/counter + cat /tmp/heartbeat/counter +' +``` + +Stateful stop: + +```bash +incus stop lab-node-01:prod-app-01 --stateful +``` + +Start (resumes from saved state): + +```bash +incus start lab-node-01:prod-app-01 +sleep 4 +incus exec lab-node-01:prod-app-01 -- cat /tmp/heartbeat/counter +``` + +Expected: `42` — the file (and entire VM state) is preserved. + +**If the restore fails** (e.g., from a `limits.cpu` mismatch), discard the +saved state: + +```bash +incus start lab-node-01:prod-app-01 --stateless +``` + + +## Section 11: Cluster Lifecycle + +### 11.1 Evacuation & Restore + +Evacuate node-02. All workloads are moved to other nodes: + +```bash +incus cluster evacuate lab-node-01:lab-node-02 --force +``` + +Check workload distribution — nothing on node-02: + +```bash +incus list lab-node-01: --columns nstL --format table +``` + +Actual behavior: VMs with `migration.stateful=true` are live-migrated +(prod-app-01 migrated to lab-node-03). Containers are stopped and moved +(prod-web-02 stopped, moved to lab-node-03, then started). The `--force` +flag skips confirmation prompts. + +**Note**: if VMs lack the `limits.cpu` range fix, use `--action stop` +instead to avoid migration failures: +```bash +incus cluster evacuate lab-node-01:lab-node-02 --force --action stop +``` + +Verify node-02 shows EVACUATED: + +```bash +incus cluster list lab-node-01: +``` + +Restore node-02 (workloads return): + +```bash +incus cluster restore lab-node-01:lab-node-02 --force +``` + +Verify all workloads are back: + +```bash +incus list lab-node-01: --columns nstL --format table +incus cluster list lab-node-01: +``` + +All nodes should show ONLINE. + +### 11.2 Node Failure Simulation + +A Proxmox hard-stop on a VM simulates a crash. The Incus cluster heartbeat +detects the failure in ~40 seconds. After the node is restarted: + +1. The node auto-rejoins the cluster (~60s) +2. Containers auto-start +3. VMs that were running resume + +**Procedure** (document only — do not execute while OVN is running unless +you can tolerate temporary network disruption): + +```bash +# Simulate crash: hard-stop via Proxmox API +# curl -s -k -X POST "https://192.168.1.29:8006/api2/json/nodes/pve/qemu/912/status/stop" ... + +# Wait for heartbeat detection (~40s) +# incus cluster list lab-node-01: +# → lab-node-02 shows OFFLINE + +# Restart via Proxmox +# curl -s -k -X POST "https://192.168.1.29:8006/api2/json/nodes/pve/qemu/912/status/start" ... + +# Wait for auto-rejoin (~60s) +# incus cluster list lab-node-01: +# → lab-node-02 shows ONLINE +``` + +### 11.3 Node Replacement + +Full procedure: evacuate a node, remove it from the cluster, destroy the +VM, deploy a fresh node, and join it back. This tests the complete lifecycle. + +**Step 1: Evacuate node-03:** + +```bash +incus cluster evacuate lab-node-01:lab-node-03 --force --action stop +``` + +**Step 2: Remove from cluster:** + +```bash +printf "yes\n" | incus cluster remove lab-node-01:lab-node-03 --force +``` + +**Note**: `incus cluster remove` prompts "Are you really sure?" even with +`--force`. The `printf` pipes `yes` for automation. + +**Step 3: Clean up the remote:** + +```bash +incus remote remove lab-node-03 +``` + +**Step 4: Destroy and redeploy the VM.** Use `incusos-proxmox` to destroy +just node-03 and redeploy it. The simplest approach: create a single-VM +config or use the replacement config pattern: + +```bash +# Destroy just node-03 via Proxmox API (VMID 913) +# Then redeploy with incusos-proxmox using a config that only defines node-03 +``` + +Alternatively, if you have a `lab-replace.yaml` config for the replacement +node: + +```bash +./incusos-proxmox --yes examples/lab-replace.yaml +``` + +**Step 5: Join the fresh node to the cluster:** + +```bash +# Set specific IP +incus config set lab-node-03: core.https_address 192.168.102.113:8443 + +# Generate join token +incus cluster add lab-node-01:lab-node-03 + +# Join +printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join lab-node-01: lab-node-03: + +# Fix remote +incus remote remove lab-node-03 +incus remote add lab-node-03 https://192.168.102.113:8443 --accept-certificate +``` + +**Step 6: Re-enable OVN on the replacement node:** + +```bash +incus query lab-node-03:/os/1.0/services/ovn --request PUT --data '{ + "config": { + "database": "tcp:192.168.102.111:6642", + "enabled": true, + "tunnel_address": "192.168.102.113", + "tunnel_protocol": "geneve" + }, + "state": {} +}' + +incus cluster role add lab-node-01:lab-node-03 ovn-chassis +``` + +**Step 7: Verify:** + +```bash +incus cluster list lab-node-01: +``` + +All 3 nodes should be ONLINE with `ovn-chassis` role. + +### 11.4 Cluster Rebalancing + +Enable automatic workload rebalancing. When a new node joins (or workloads +are unevenly distributed), Incus redistributes VMs: + +```bash +incus config set lab-node-01: cluster.rebalance.interval=1 +incus config set lab-node-01: cluster.rebalance.threshold=10 +incus config set lab-node-01: cluster.rebalance.batch=2 +incus config set lab-node-01: cluster.rebalance.cooldown=5m +``` + +**Important**: only VMs with `migration.stateful=true` are rebalanced. +Containers are NOT auto-rebalanced. + +Monitor rebalancing: + +```bash +incus list lab-node-01: --columns nstL --format table +``` + +Disable rebalancing when done testing: + +```bash +incus config unset lab-node-01: cluster.rebalance.interval +incus config unset lab-node-01: cluster.rebalance.threshold +``` + + +## Section 12: OC Dashboard + +**Important**: Switch to the OC remote first. The OC CLI does not support +`remote:` suffix syntax: + +```bash +operations-center remote switch oc-lab +``` + +### OC Server Information + +```bash +operations-center admin os show +``` + +Actual output: + +``` ++-------------------+----------------------------+ +| PROPERTY | VALUE | ++-------------------+----------------------------+ +| hostname | lab-oc | +| os_version | 202602210344 | +| kernel | 6.12.13 | +| architecture | x86_64 | +| uptime | 4390 | +| addresses | 192.168.102.110/22 (mgmt) | +| | fd42:...:1 (incusbr0) | +| storage_disks | /dev/sda (53.7GB, QEMU) | +| storage_pools | local (zfs, /dev/sda4) | ++-------------------+----------------------------+ +``` + +### Application Status + +```bash +operations-center admin os application list +``` + +Actual output: + +``` ++--------------------+---------+ +| NAME | STATUS | ++--------------------+---------+ +| operations-center | running | ++--------------------+---------+ +``` + +### Service Status + +```bash +operations-center admin os service list +``` + +Actual output: + +``` ++----------+---------+ +| NAME | ENABLED | ++----------+---------+ +| ovn | false | +| syslog | false | +| fan | false | +| bgp | false | +| dns | false | +| metricsA | false | +| metricsB | false | ++----------+---------+ +``` + +### Web UI + +The OC web UI at `https://192.168.102.110:8443/ui/` provides: + +- **Dashboard**: server overview with resource utilization +- **Updates**: available IncusOS updates +- **Provisioning**: token management (for OC-provisioned deployments) +- **System**: OC configuration and certificates + +**Limitation**: OC deployed with a standard ISO cannot manage the Incus +cluster nodes. The nodes are independent — they were deployed with a +standard IncusOS ISO, not an OC-provisioned one. For full OC node +management (cluster orchestration, application deployment, monitoring), +nodes must boot from an OC-provisioned ISO. + +See [Operations Center Guide](operations-center-guide.md) for the full +hybrid deployment workflow with OC-provisioned ISOs. + + +## Section 13: Cleanup + +### Delete All Workloads + +```bash +# Delete containers +for c in prod-web-01 prod-web-02 prod-api-01; do + incus delete lab-node-01:$c --force +done + +# Delete VMs +for vm in prod-db-01 prod-app-01; do + incus delete lab-node-01:$vm --force +done +``` + +### Remove OVN Networks + +```bash +# Delete OVN networks +incus network delete lab-node-01:net-prod +incus network delete lab-node-01:net-isolated 2>/dev/null || true + +# Delete UPLINK +incus network delete lab-node-01:UPLINK +``` + +### Remove OVN Control Plane + +```bash +incus delete lab-node-01:ovn-central --force +``` + +### Disable OVN Services + +```bash +for node in lab-node-01 lab-node-02 lab-node-03; do + incus query "$node":/os/1.0/services/ovn --request PUT --data '{ + "config": { + "enabled": false + }, + "state": {} + }' +done +``` + +### Infrastructure Options + +**Keep infrastructure** (stop VMs, keep on disk for later): + +```bash +./incusos-proxmox --lab-down examples/lab-production.yaml +``` + +Restart later with: + +```bash +./incusos-proxmox --lab-up examples/lab-production.yaml +``` + +**Full teardown** (destroy all VMs, remove ISOs, remotes, cache): + +```bash +./incusos-proxmox --cleanup --deep examples/lab-production.yaml +``` + + +## Section 14: Verification Checklist + +| # | Check | Command | Expected | +|---|-------|---------|----------| +| 1 | All VMs running | `incusos-proxmox --status examples/lab-production.yaml` | 4 VMs running, port 8443 open | +| 2 | Scrub schedule healthy | `incus query lab-node-01:/os/1.0/system/storage` | `scrub_schedule: "0 4 * * 0"` | +| 3 | OC accessible | `operations-center remote switch oc-lab && operations-center admin os show` | Shows version, uptime | +| 4 | Cluster formed | `incus cluster list lab-node-01:` | 3 nodes ONLINE | +| 5 | Storage pool | `incus storage list lab-node-01:` | `local` pool on all members | +| 6 | Bridge isolation | Ping cross-node on incusbr0 | 100% loss (expected) | +| 7 | OVN connectivity | Ping cross-node on net-prod | 0% loss | +| 8 | Internet via OVN | `ping 1.1.1.1` from OVN container | 0% loss | +| 9 | Network isolation | Ping net-prod → net-isolated | 100% loss (expected) | +| 10 | Network peering | Peer + ping cross-network | 0% loss, TTL=62 | +| 11 | Load balancer | `curl http://192.168.103.200` | Backend response | +| 12 | Network forward | `curl http://192.168.103.201:8080` | API response | +| 13 | DNS resolution | `dig prod-web-02.incus` from container | Resolves to 10.10.10.x | +| 14 | VM live migration | `incus move` VM between nodes | State preserved | +| 15 | Cluster evacuation | `incus cluster evacuate` + `restore` | Workloads moved and returned | +| 16 | Stateful stop/start | `incus stop --stateful` + `start` | VM state preserved | + + +## Section 15: Quick Reference + +### Cluster Command Syntax + +| Command | Arguments | Notes | +|---------|-----------|-------| +| `incus cluster enable` | `remote: member-name` | TWO args (space between) | +| `incus cluster add` | `remote:member-name` | ONE arg (no space) | +| `incus cluster join` | `init-remote: joining-remote:` | TWO args (space between) | +| `incus cluster remove` | `remote:member-name --force` | ONE arg; prompts even with `--force` | +| `incus cluster evacuate` | `remote:member-name` | ONE arg (no space) | +| `incus cluster restore` | `remote:member-name` | ONE arg (no space) | +| `incus config set` | `remote: key value` | Remote with trailing colon + space | +| `incus storage show` | `remote:pool` | ONE arg (no space) | +| `incus storage show` | `remote:pool --target member` | `--target` for member-specific | + +### OVN Setup Cheat Sheet + +```bash +# 1. Deploy OVN container +incus launch images:debian/12 REMOTE:ovn-central --target NODE +incus exec REMOTE:ovn-central -- apt-get install -y ovn-central ovn-host +incus exec REMOTE:ovn-central -- ovn-nbctl set-connection ptcp:6641:0.0.0.0 +incus exec REMOTE:ovn-central -- ovn-sbctl set-connection ptcp:6642:0.0.0.0 +incus config device add REMOTE:ovn-central nb-proxy proxy listen=tcp:HOST_IP:6641 connect=tcp:127.0.0.1:6641 +incus config device add REMOTE:ovn-central sb-proxy proxy listen=tcp:HOST_IP:6642 connect=tcp:127.0.0.1:6642 + +# 2. Enable OVN on each IncusOS node +incus query NODE:/os/1.0/services/ovn --request PUT --data '{"config":{"database":"tcp:HOST_IP:6642","enabled":true,"tunnel_address":"NODE_IP","tunnel_protocol":"geneve"},"state":{}}' + +# 3. Configure Incus +incus config set REMOTE: network.ovn.northbound_connection tcp:HOST_IP:6641 +incus cluster role add REMOTE:MEMBER ovn-chassis # for each member + +# 4. Create UPLINK (per-member then cluster-wide) +incus network create REMOTE:UPLINK --type physical --target MEMBER parent=mgmt # each member +incus network create REMOTE:UPLINK --type physical ipv4.ovn.ranges=RANGE ipv4.gateway=GW/PREFIX + +# 5. Create OVN network +incus network create REMOTE:net-name --type=ovn network=UPLINK ipv4.address=SUBNET ipv4.nat=true +``` + +### Migration Readiness Checklist + +| Setting | Value | Why | +|---------|-------|-----| +| `limits.cpu` | Range (e.g., `0-1`) | Fixed QEMU topology across hosts | +| `migration.stateful` | `true` | Enables live migration | +| `root size.state` | `2GiB` (or `4GiB` for 3-4 vCPUs) | Space for memory state file | + +Configure while VM is **stopped**: + +```bash +incus stop REMOTE:VM +incus config set REMOTE:VM limits.cpu=0-1 +incus config set REMOTE:VM migration.stateful=true +incus config device set REMOTE:VM root size.state=2GiB +incus start REMOTE:VM +``` + +### Troubleshooting + +| Symptom | Cause | Fix | +|---------|-------|-----| +| Port 8443 not reachable after boot | Boot still in progress or crontab bug | Wait 180s; check scrub_schedule via API | +| `scrub_schedule` empty | Crontab race condition | `incusos-proxmox --status` auto-heals | +| `Missing section footer for ICH9LPC` on migration | `limits.cpu` set as integer | Set as range: `limits.cpu=0-1` | +| `VM agent isn't currently running` after migration | Agent reconnecting | `sleep 4` after migration | +| `db.sock not found` on OVN config | OVN service not enabled on IncusOS | Enable via `/os/1.0/services/ovn` API | +| Cross-node ping fails (bridge) | Bridge networks are node-local | Use OVN network instead | +| `zfs load-key: Raw key too short` | TPM corruption from premature VM stop | Destroy and redeploy VM | +| Cluster join fails with "pool already exists" | `apply_defaults: true` on joining node | Use `apply_defaults: false` or run 8-command cleanup | +| OC cannot manage cluster nodes | Nodes deployed with standard ISO | Use OC-provisioned ISO for full integration | +| `CPUID vnmi` warning during migration | Cosmetic QEMU check | Safe to ignore | +| "Parent interface 'ens18' not found" | IncusOS names its NIC `mgmt` | Use `parent=mgmt` for UPLINK network | +| "Invalid target address" on LB backend | Backend needs IP, not instance name | Use instance IP address (e.g., `10.10.10.2`) | +| OC CLI "Invalid number of arguments" | OC CLI doesn't support `remote:` suffix | Use `operations-center remote switch NAME` first | +| Container placed on wrong node | Cluster auto-schedules without `--target` | Use `--target NODE` for explicit placement |