Add production lab guide with validated output from real deployment

Comprehensive 1775-line guide (notes/production-lab-guide.md) covering
full lab lifecycle: OC dashboard, 3-node manual clustering, OVN overlay
networking, mixed workloads, live migration, network security, load
balancers, and cluster evacuation/restore. Every command was executed
on real infrastructure (2026-02-22) and output captured.

Key findings incorporated:
- IncusOS NIC is named `mgmt`, not `ens18` (UPLINK parent)
- OC CLI requires `remote switch`, not `remote:` suffix syntax
- LB/forward backends need IP addresses, not instance names
- Cluster auto-schedules without --target flag
- Migration ~7s at ~140 MB/s with heartbeat continuity

Also: lab-production.yaml config, lab-oc.yaml apply_defaults fix,
CLAUDE.md and networking-guide.md updates for mgmt/OC/LB findings.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Maarten 2026-02-22 23:46:10 +01:00
parent 088baf6807
commit d8f2a067c4
5 changed files with 1872 additions and 5 deletions

View File

@ -441,6 +441,10 @@ incu-contrib/
imported into the browser. Web UI is a React SPA at `/ui/`. imported into the browser. Web UI is a React SPA at `/ui/`.
- **OC is under active development** (v0.3.0). Commands and APIs may change. - **OC is under active development** (v0.3.0). Commands and APIs may change.
Use `operations-center --version` (not `version` subcommand) to check. Use `operations-center --version` (not `version` subcommand) to check.
- **OC CLI does NOT support `remote:` suffix syntax**: unlike the Incus CLI,
the OC CLI requires `operations-center remote switch NAME` before running
commands. `operations-center admin os show oc-lab:` fails with "Invalid
number of arguments".
- **v0.3.0 new commands**: `provisioning channel` (add/list/show), - **v0.3.0 new commands**: `provisioning channel` (add/list/show),
`cluster update/update-certificate/rename`, expanded `inventory` `cluster update/update-certificate/rename`, expanded `inventory`
(network-acl, address-sets, load-balancers, peers, zones, integrations, (network-acl, address-sets, load-balancers, peers, zones, integrations,
@ -839,7 +843,8 @@ sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh -o StrictHostKeyChecking=no \
4. `incus cluster role add <remote>:<member> ovn-chassis` (all nodes) 4. `incus cluster role add <remote>:<member> ovn-chassis` (all nodes)
5. Create physical uplink network (two-step cluster pattern) 5. Create physical uplink network (two-step cluster pattern)
6. Create OVN network with `--type=ovn network=UPLINK` 6. Create OVN network with `--type=ovn network=UPLINK`
- **Physical uplink network**: uses `parent=ens18` (IncusOS default NIC), - **Physical uplink network**: uses `parent=mgmt` (IncusOS management NIC —
NOT `ens18`, which is the underlying device name but not exposed to Incus).
`ipv4.ovn.ranges` reserves LAN IPs for OVN router external addresses, `ipv4.ovn.ranges` reserves LAN IPs for OVN router external addresses,
`ipv4.gateway` is the LAN gateway in CIDR format. `ipv4.gateway` is the LAN gateway in CIDR format.
- **OVN network isolation**: multiple OVN networks are fully isolated. - **OVN network isolation**: multiple OVN networks are fully isolated.
@ -849,6 +854,10 @@ sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh -o StrictHostKeyChecking=no \
ACLs (per-source blocking), network peering, L4 load balancers (connection- ACLs (per-source blocking), network peering, L4 load balancers (connection-
based hashing, not round-robin), network forwards (port forwarding to LAN based hashing, not round-robin), network forwards (port forwarding to LAN
IPs), DNS resolution (per-network, hostname.incus domain). IPs), DNS resolution (per-network, hostname.incus domain).
- **LB/forward backends require IP addresses**: `incus network load-balancer
backend add` and `incus network forward port add` require the target
instance's IP address, NOT its name. Using instance names fails with
"Invalid target address".
- See `notes/networking-guide.md` for full tutorial with test results. - See `notes/networking-guide.md` for full tutorial with test results.
### Migration into Incus ### Migration into Incus

View File

@ -3,6 +3,11 @@
# Deploys 1 Operations Center instance and 3 Incus nodes. # Deploys 1 Operations Center instance and 3 Incus nodes.
# Uses VMID range 910+ to avoid collisions with lab-cluster.yaml (900+). # Uses VMID range 910+ to avoid collisions with lab-cluster.yaml (900+).
# #
# The init node (lab-node-01) uses apply_defaults: true for storage pool
# and network bridge. Joining nodes use apply_defaults: false -- the cluster
# join process creates member-specific pool entries. This follows the upstream
# IncusOS recommendation and avoids the 8-command cleanup per joining node.
#
# Workflow: # Workflow:
# 1. Deploy with incusos-proxmox # 1. Deploy with incusos-proxmox
# 2. Access OC web UI (import client.pfx into browser) # 2. Access OC web UI (import client.pfx into browser)
@ -33,7 +38,7 @@ vms:
- name: lab-node-01 - name: lab-node-01
app: incus app: incus
apply_defaults: true apply_defaults: true # init node: needs storage pool + network
cores: 4 cores: 4
memory: 8192 memory: 8192
disk: 64 disk: 64
@ -41,10 +46,14 @@ vms:
- name: lab-node-02 - name: lab-node-02
app: incus app: incus
apply_defaults: true apply_defaults: false # joining node: cluster join creates pool entry
cores: 4
memory: 8192
ip: 192.168.102.112/22 ip: 192.168.102.112/22
- name: lab-node-03 - name: lab-node-03
app: incus app: incus
apply_defaults: true apply_defaults: false # joining node: cluster join creates pool entry
cores: 4
memory: 8192
ip: 192.168.102.113/22 ip: 192.168.102.113/22

View File

@ -0,0 +1,71 @@
# lab-production.yaml - Production-quality home lab (OC + 3-node cluster)
#
# Deploys 1 Operations Center instance and 3 Incus nodes for manual clustering.
# Uses VMID range 910-913 (same as lab-oc.yaml -- do not run both simultaneously).
#
# Architecture:
# lab-oc (910) - Operations Center dashboard and monitoring
# lab-node-01 (911) - Cluster init node + OVN control plane host (64G disk)
# lab-node-02 (912) - Cluster member
# lab-node-03 (913) - Cluster member
#
# Strategy: Hybrid OC + Manual Clustering
# - OC deployed with standard ISO (dashboard/monitoring role)
# - 3 nodes deployed with standard ISO, static IPs on VLAN 69
# - Cluster formed manually via incus CLI (proven reliable)
# - OVN overlay networking configured post-clustering
# - Mixed workloads: containers + VMs with live migration
#
# Joining nodes use apply_defaults: false -- the cluster join process creates
# member-specific pool entries. This follows the upstream IncusOS recommendation
# and avoids the 8-command cleanup per joining node.
#
# RAM budget: 28 GiB of 64 GiB (44% utilization)
# OVN external IPs: 192.168.103.200-210 (excluded from DHCP)
#
# Usage:
# incusos-proxmox --dry-run examples/lab-production.yaml # Preview
# incusos-proxmox --yes examples/lab-production.yaml # Deploy
# incusos-proxmox --status examples/lab-production.yaml # Check status
# incusos-proxmox --cleanup examples/lab-production.yaml # Tear down VMs
# incusos-proxmox --cleanup --deep examples/lab-production.yaml # Full cleanup
#
# After deployment, follow notes/production-lab-guide.md for:
# - Cluster formation (section 4)
# - OVN networking (section 6)
# - Workload deployment (sections 7-9)
# - Live migration testing (section 10)
# - Cluster lifecycle (section 11)
#
# Connection settings from proxmox.yaml (see examples/proxmox.yaml.example).
# Add a proxmox: section here to override per-deployment.
defaults:
cores: 4
memory: 8192
disk: 50
start_vmid: 910
vms:
- name: lab-oc
app: operations-center
apply_defaults: true
cores: 2
memory: 4096
ip: 192.168.102.110/22
- name: lab-node-01
app: incus
apply_defaults: true # init node: needs storage pool + network
disk: 64 # extra space for OVN control plane container
ip: 192.168.102.111/22
- name: lab-node-02
app: incus
apply_defaults: false # joining node: cluster join creates pool entry
ip: 192.168.102.112/22
- name: lab-node-03
app: incus
apply_defaults: false # joining node: cluster join creates pool entry
ip: 192.168.102.113/22

View File

@ -380,7 +380,10 @@ incus network create net-node-01:UPLINK --type=physical \
dns.nameservers=192.168.1.1 dns.nameservers=192.168.1.1
``` ```
- `parent=ens18`: the node's physical NIC (IncusOS default on Proxmox) - `parent=ens18`: the node's physical NIC. **Note**: newer IncusOS versions
name this interface `mgmt`. If `ens18` fails with "Parent interface not
found", use `parent=mgmt` instead. Check with:
`incus query REMOTE:/os/1.0/system/network`
- `ipv4.ovn.ranges`: pool of LAN IPs for OVN router external addresses - `ipv4.ovn.ranges`: pool of LAN IPs for OVN router external addresses
- `ipv4.gateway`: LAN gateway in CIDR format - `ipv4.gateway`: LAN gateway in CIDR format

File diff suppressed because it is too large Load Diff