# Production Home Lab Guide

Build a production-quality Incus home lab from scratch: Operations Center
dashboard, 3-node cluster with OVN overlay networking, mixed container/VM
workloads, live migration, network security, load balancers, and cluster
lifecycle management.

All commands and output in this guide are from an actual deployment on
2026-02-22. Tested on Proxmox VE 9.1.5, IncusOS 202602210344, Incus
client 6.21, Operations Center v0.3.0.


## Section 0: Architecture Overview

### Network Topology

```mermaid
flowchart TD
    vlan(("VLAN 69<br/>192.168.100.0/22"))

    subgraph mgmt["Management"]
        oc["lab-oc<br/>VMID 910 · .110<br/>OC server"]
    end

    subgraph cluster["Incus Cluster"]
        n1["lab-node-01<br/>VMID 911 · .111<br/>init + ovn-central"]
        n2["lab-node-02<br/>VMID 912 · .112"]
        n3["lab-node-03<br/>VMID 913 · .113"]
    end

    subgraph networks["OVN Networks"]
        prod("net-prod<br/>10.10.10.0/24")
        iso("net-isolated<br/>10.10.20.0/24")
        uplink("UPLINK<br/>.103.200-210")
    end

    vlan --- mgmt & cluster
    cluster -->|"Geneve tunnels"| networks

    classDef nodeClass fill:#009E73,color:#fff,stroke:#007a5e
    classDef mgmtClass fill:#CC79A7,color:#fff,stroke:#a36088
    classDef networkClass fill:#0072B2,color:#fff,stroke:#005a8e

    class n1,n2,n3 nodeClass
    class oc mgmtClass
    class prod,iso,uplink,vlan networkClass

    style mgmt fill:#f5e6f0,stroke:#CC79A7
    style cluster fill:#e6f5f0,stroke:#009E73
    style networks fill:#e0eef8,stroke:#0072B2
```

### Infrastructure

| Component | VMID | IP | Cores | RAM | Disk | Role |
|-----------|------|-----|-------|-----|------|------|
| lab-oc | 910 | 192.168.102.110/22 | 2 | 4 GiB | 50G | Operations Center |
| lab-node-01 | 911 | 192.168.102.111/22 | 4 | 8 GiB | 64G | Cluster init + OVN host |
| lab-node-02 | 912 | 192.168.102.112/22 | 4 | 8 GiB | 50G | Cluster member |
| lab-node-03 | 913 | 192.168.102.113/22 | 4 | 8 GiB | 50G | Cluster member |

**RAM budget**: 28 GiB of 64 GiB (44% utilization). Leaves headroom for
workloads inside the VMs and other labs on the host.

**OVN IP allocation**: 192.168.103.200-210 reserved for OVN external
addresses (router IPs, load balancer VIPs, network forwards). These must
be excluded from your DHCP server's range.

### Decision Rationale

**Why manual clustering instead of OC `provisioning cluster add`?**
OC v0.3.0's `provisioning cluster add` has a `needs update: false` blocker
that can stall indefinitely. Manual clustering via `incus` CLI is proven
reliable. OC still provides value as a monitoring dashboard.

**Why OVN?** Bridge networks are node-local — instances on different nodes
cannot communicate. OVN provides transparent cross-node L2 overlay with
sub-millisecond latency (~0.1-0.8ms), network isolation, ACLs, load
balancers, and network forwards.

**Why VLAN 69?** Isolates lab traffic from the production LAN. All VMs share
VLAN 69 (subnet 192.168.100.0/22). The VLAN tag is set at the Proxmox VM
level — IncusOS and workloads are unaware of it.

### Cross-References

This guide brings together techniques from the deep-dive guides:

- [Clustering Guide](clustering-guide.md) — cluster formation, migration, lifecycle
- [Networking Guide](networking-guide.md) — OVN setup, ACLs, peering, LBs
- [Operations Center Guide](operations-center-guide.md) — OC provisioning, CLI, web UI
- [Migration Guide](migration-guide.md) — importing VMs from other hypervisors


## Section 1: Prerequisites

### Required Tools

Verify all tools are available before starting:

```bash
incus version
operations-center --version
bash --version | head -1
python3 --version
jq --version
curl --version | head -1
genisoimage --version 2>&1 | head -1
```

**Minimum versions**: Incus client 6.3+ (for `remote get-client-certificate`
fallback, though scripts read `~/.config/incus/client.crt` directly),
Operations Center v0.3.0+.

### Proxmox Configuration

Your `incusos/proxmox.yaml` should contain:

```yaml
host: 192.168.1.29
method: api
api_token_id: automation@pve!deploy
node: pve
storage: local-zfs
iso_storage: local
bridge: vmbr0
vlan: 69
gateway: 192.168.100.1
dns: 192.168.100.1
pool: IncusLab
```

The `env` file at the repository root must export `PROXMOX_TOKEN_SECRET`.
Scripts auto-discover it — no manual sourcing needed.

### Client Certificates

Incus client certificates are used for both Incus and OC connections:

```bash
# Verify cert exists (auto-generated on first incus command)
ls -la ~/.config/incus/client.crt ~/.config/incus/client.key
```

For OC web UI browser access, generate a PKCS#12 bundle:

```bash
openssl pkcs12 -export \
    -out ~/.config/incus/client.pfx \
    -inkey ~/.config/incus/client.key \
    -in ~/.config/incus/client.crt \
    -name "Incus Client"
```

Import `client.pfx` into your browser's certificate store (Firefox:
Settings → Privacy & Security → View Certificates → Import).

### Doctor Check

Run the environment check to verify everything is in order:

```bash
cd incusos
./incusos-proxmox --doctor
```

Expected output includes tool versions, IncusOS CDN reachability,
`proxmox.yaml` discovery, and Proxmox API connectivity.


## Section 2: Deploy Infrastructure

### Configuration File

The lab uses `incusos/examples/lab-production.yaml`:

```yaml
defaults:
  cores: 4
  memory: 8192
  disk: 50
  start_vmid: 910

vms:
  - name: lab-oc
    app: operations-center
    apply_defaults: true
    cores: 2
    memory: 4096
    ip: 192.168.102.110/22

  - name: lab-node-01
    app: incus
    apply_defaults: true          # init node: needs storage pool + network
    disk: 64                      # extra space for OVN control plane container
    ip: 192.168.102.111/22

  - name: lab-node-02
    app: incus
    apply_defaults: false         # joining node: cluster join creates pool entry
    ip: 192.168.102.112/22

  - name: lab-node-03
    app: incus
    apply_defaults: false         # joining node: cluster join creates pool entry
    ip: 192.168.102.113/22
```

**Key decisions**: node-01 has `apply_defaults: true` (cluster init needs
storage pool and network bridge). Nodes 02 and 03 have `apply_defaults: false`
(the cluster join process creates member-specific entries). node-01 gets
64 GiB disk for the OVN control plane container.

### Dry Run

Preview the deployment without making any changes:

```bash
./incusos-proxmox --dry-run examples/lab-production.yaml
```

This shows: ISO download plan, seed generation commands, VM creation
parameters, and the full install sequence for each VM.

### Deploy

Deploy all 4 VMs:

```bash
./incusos-proxmox --yes examples/lab-production.yaml
```

The deploy takes ~5-8 minutes:

1. Downloads the latest IncusOS ISO (if not cached)
2. Generates per-VM seed ISOs with static IP, hostname, certificates
3. Creates VMs on Proxmox with UEFI, TPM, VirtIO settings
4. Boots each VM with ISO + seed, monitors installation via blockstat
5. Detects install completion (876 MiB written, then idle)
6. Stops VMs, removes install media (ISO + seed)
7. Starts VMs from disk, waits for port 8443 (up to 180s)
8. Auto-heals scrub_schedule via IncusOS REST API
9. Configures `incus` remotes for each Incus node

### Verify Deployment

Check deployment status:

```bash
./incusos-proxmox --status examples/lab-production.yaml
```

Expected output shows each VM with Proxmox state (running), network
(static IP reachable), port 8443 (open), and incus remote (configured).

### Verify Scrub Schedule

Confirm the crontab bug fix is effective on all Incus nodes:

```bash
for node in lab-node-01 lab-node-02 lab-node-03; do
    echo -n "$node scrub_schedule: "
    incus query "$node":/os/1.0/system/storage | python3 -c \
        "import sys,json; print(json.load(sys.stdin).get('config',{}).get('scrub_schedule','EMPTY'))"
done
```

Expected output — all nodes show `0 4 * * 0`:

```
lab-node-01 scrub_schedule: 0 4 * * 0
lab-node-02 scrub_schedule: 0 4 * * 0
lab-node-03 scrub_schedule: 0 4 * * 0
```

If any node shows `EMPTY`, the crontab bug hit. Run:
```bash
./incusos-proxmox --status examples/lab-production.yaml
```
The status check includes automatic scrub_schedule healing.


## Section 3: Operations Center Setup

### Add OC Remote

```bash
operations-center remote add oc-lab https://192.168.102.110:8443 --auth-type tls
```

Accept the certificate fingerprint when prompted.

**Important**: The OC CLI does **not** support the `remote:` suffix syntax
that the Incus CLI uses. Instead, switch to the remote first, then run
commands without a remote suffix:

```bash
operations-center remote switch oc-lab
```

### Verify OC

```bash
operations-center admin os show
```

Actual output:

```
+-------------------+----------------------------+
|     PROPERTY      |           VALUE            |
+-------------------+----------------------------+
| hostname          | lab-oc                     |
| os_version        | 202602210344               |
| kernel            | 6.12.13                    |
| architecture      | x86_64                     |
| uptime            | 4390                       |
| addresses         | 192.168.102.110/22 (mgmt)  |
|                   | fd42:...:1 (incusbr0)      |
| storage_disks     | /dev/sda (53.7GB, QEMU)    |
| storage_pools     | local (zfs, /dev/sda4)     |
+-------------------+----------------------------+
```

### Check Application Status

```bash
operations-center admin os application list
```

Actual output:

```
+--------------------+---------+
|        NAME        | STATUS  |
+--------------------+---------+
| operations-center  | running |
+--------------------+---------+
```

### Check for Updates

```bash
operations-center provisioning update list
```

Shows available IncusOS updates. Updates can be applied via the OC web UI
or CLI.

### Service Status

```bash
operations-center admin os service list
```

Actual output:

```
+----------+---------+
|   NAME   | ENABLED |
+----------+---------+
| ovn      | false   |
| syslog   | false   |
| fan      | false   |
| bgp      | false   |
| dns      | false   |
| metricsA | false   |
| metricsB | false   |
+----------+---------+
```

### Web UI Access

Open `https://192.168.102.110:8443/ui/` in your browser. You need the
PKCS#12 client certificate imported (see Section 1). The web UI provides
a dashboard view of the OC server. After adding Incus nodes to OC
(optional), the dashboard shows cluster health.

**Note**: OC deployed with a standard ISO acts as a monitoring dashboard.
For full OC node management (provisioning, cluster orchestration), nodes
must boot from an OC-provisioned ISO. See
[Operations Center Guide](operations-center-guide.md) for the full hybrid
workflow.


## Section 4: Cluster Formation

### 4.1 Set Specific IP Addresses

IncusOS nodes default to `core.https_address: :8443` (wildcard). Clustering
requires specific routable IPs so nodes can address each other.

```bash
incus config set lab-node-01: core.https_address 192.168.102.111:8443
incus config set lab-node-02: core.https_address 192.168.102.112:8443
incus config set lab-node-03: core.https_address 192.168.102.113:8443
```

Verify on each node:

```bash
incus config get lab-node-01: core.https_address
incus config get lab-node-02: core.https_address
incus config get lab-node-03: core.https_address
```

Each should return `IP:8443`.

### 4.2 Enable Clustering on Init Node

```bash
incus cluster enable lab-node-01: lab-node-01
```

**Note the syntax**: TWO arguments — `lab-node-01:` (remote with trailing
colon) and `lab-node-01` (member name). This is NOT `lab-node-01:lab-node-01`.

### 4.3 Fix Init Node Remote

Enabling clustering regenerates the TLS certificate. The new cert may only
have SANs for `127.0.0.1` and `::1`, breaking the remote.

```bash
incus remote switch local
incus remote remove lab-node-01
incus remote add lab-node-01 https://192.168.102.111:8443 --accept-certificate
```

Verify:

```bash
incus cluster list lab-node-01:
```

Expected output:

```
+--------------+----------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
|     NAME     |            URL             |      ROLES       | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATE  |      MESSAGE      |
+--------------+----------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| lab-node-01  | https://192.168.102.111:8443| database-leader | x86_64       | default        |             | ONLINE | Fully operational |
|              |                            | database         |              |                |             |        |                   |
+--------------+----------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
```

### 4.4 Join Node-02

Generate a join token on the init node:

```bash
incus cluster add lab-node-01:lab-node-02
```

This outputs a long base64 token. Use it immediately — tokens expire.

Join node-02 to the cluster (automated, non-interactive):

```bash
printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join lab-node-01: lab-node-02:
```

The five prompts answered by `printf`:
1. IP address → accept default (node's IP)
2. Member name → accept default (matches token)
3. "All existing data is lost" → `yes`
4. `source` for storage pool "local" → `local/incus`
5. `zfs.pool_name` for pool "local" → `local/incus`

No storage/network cleanup needed — `apply_defaults: false` means node-02
has no pre-existing Incus storage pool or network.

Fix the remote after join (new cluster cert):

```bash
incus remote remove lab-node-02
incus remote add lab-node-02 https://192.168.102.112:8443 --accept-certificate
```

### 4.5 Join Node-03

Same procedure:

```bash
incus cluster add lab-node-01:lab-node-03
printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join lab-node-01: lab-node-03:
```

Fix the remote:

```bash
incus remote remove lab-node-03
incus remote add lab-node-03 https://192.168.102.113:8443 --accept-certificate
```

### 4.6 Verify Cluster

```bash
incus cluster list lab-node-01:
```

Expected output — 3 nodes, all ONLINE:

```
+--------------+----------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
|     NAME     |            URL             |      ROLES       | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATE  |      MESSAGE      |
+--------------+----------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| lab-node-01  | https://192.168.102.111:8443| database-leader | x86_64       | default        |             | ONLINE | Fully operational |
|              |                            | database         |              |                |             |        |                   |
+--------------+----------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| lab-node-02  | https://192.168.102.112:8443| database         | x86_64       | default        |             | ONLINE | Fully operational |
+--------------+----------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| lab-node-03  | https://192.168.102.113:8443| database         | x86_64       | default        |             | ONLINE | Fully operational |
+--------------+----------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
```

Verify storage pool exists on all members:

```bash
incus storage show lab-node-01:local
incus storage show lab-node-01:local --target lab-node-02
incus storage show lab-node-01:local --target lab-node-03
```

Verify the default network:

```bash
incus network list lab-node-01:
```


## Section 5: Bridge Networking Baseline

Before setting up OVN, establish the baseline: bridge networks are
node-local. This demonstrates why OVN is needed.

### Same-Node Communication

Launch 2 containers on the same node. **Important**: use `--target` to force
placement — without it, the cluster scheduler may place containers on
different nodes automatically:

```bash
incus launch images:debian/12 lab-node-01:test-bridge-a --target lab-node-01
incus launch images:debian/12 lab-node-01:test-bridge-b --target lab-node-01
```

Wait for them to get IPs:

```bash
incus list lab-node-01: --columns ns4 --format csv | grep test-bridge
```

Ping between them:

```bash
IP_B=$(incus list lab-node-01:test-bridge-b --columns 4 --format csv | cut -d' ' -f1)
incus exec lab-node-01:test-bridge-a -- ping -c 3 "$IP_B"
```

Actual result: 0% packet loss, ~0.024ms latency. Same bridge, same node — works.

### Cross-Node Communication (Fails)

Launch a container on a different node:

```bash
incus launch images:debian/12 lab-node-01:test-bridge-c --target lab-node-02
```

Wait for IP:

```bash
incus list lab-node-01: --columns ns4 --format csv | grep test-bridge
```

Ping from node-01 to node-02:

```bash
IP_C=$(incus list lab-node-01:test-bridge-c --columns 4 --format csv | cut -d' ' -f1)
incus exec lab-node-01:test-bridge-a -- ping -c 3 -W 2 "$IP_C"
```

Actual result: **100% packet loss**. Bridge networks are node-local — there is
no L2 path between `incusbr0` on node-01 and `incusbr0` on node-02. Each
node's bridge has the same subnet (e.g., 10.251.22.1/24) but they are
separate L2 domains.

### Internet Access

NAT to the internet works from any node:

```bash
incus exec lab-node-01:test-bridge-a -- ping -c 3 1.1.1.1
```

Actual result: 0% packet loss, ~10ms latency. Each bridge provides NAT via
the host's management interface.

### Cleanup

```bash
incus delete lab-node-01:test-bridge-a --force
incus delete lab-node-01:test-bridge-b --force
incus delete lab-node-01:test-bridge-c --force
```


## Section 6: OVN Overlay Networking

OVN provides a cross-node L2 overlay using Geneve tunnels. After this
section, containers on any node can communicate transparently.

### 6.1 Deploy OVN Control Plane

Launch a Debian container on node-01 to host the OVN central services:

```bash
incus launch images:debian/12 lab-node-01:ovn-central --target lab-node-01
```

Install OVN:

```bash
incus exec lab-node-01:ovn-central -- apt-get update
incus exec lab-node-01:ovn-central -- apt-get install -y ovn-central ovn-host
```

Configure OVN to listen on all interfaces:

```bash
incus exec lab-node-01:ovn-central -- ovn-nbctl set-connection ptcp:6641:0.0.0.0
incus exec lab-node-01:ovn-central -- ovn-sbctl set-connection ptcp:6642:0.0.0.0
```

Add proxy devices to expose NB and SB ports on the host's LAN IP:

```bash
incus config device add lab-node-01:ovn-central nb-proxy proxy \
    listen=tcp:192.168.102.111:6641 connect=tcp:127.0.0.1:6641
incus config device add lab-node-01:ovn-central sb-proxy proxy \
    listen=tcp:192.168.102.111:6642 connect=tcp:127.0.0.1:6642
```

Verify the ports are reachable:

```bash
curl -s --connect-timeout 2 telnet://192.168.102.111:6641 || echo "NB port open"
curl -s --connect-timeout 2 telnet://192.168.102.111:6642 || echo "SB port open"
```

### 6.2 Enable OVN on All IncusOS Nodes

OVN services are disabled by default on IncusOS. Enable them on every node
via the IncusOS REST API. The `database` field points to the **southbound**
DB (port 6642, not 6641).

**Node-01:**

```bash
incus query lab-node-01:/os/1.0/services/ovn --request PUT --data '{
  "config": {
    "database": "tcp:192.168.102.111:6642",
    "enabled": true,
    "tunnel_address": "192.168.102.111",
    "tunnel_protocol": "geneve"
  },
  "state": {}
}'
```

**Node-02:**

```bash
incus query lab-node-02:/os/1.0/services/ovn --request PUT --data '{
  "config": {
    "database": "tcp:192.168.102.111:6642",
    "enabled": true,
    "tunnel_address": "192.168.102.112",
    "tunnel_protocol": "geneve"
  },
  "state": {}
}'
```

**Node-03:**

```bash
incus query lab-node-03:/os/1.0/services/ovn --request PUT --data '{
  "config": {
    "database": "tcp:192.168.102.111:6642",
    "enabled": true,
    "tunnel_address": "192.168.102.113",
    "tunnel_protocol": "geneve"
  },
  "state": {}
}'
```

Each call should return `{}` on success.

### 6.3 Configure Incus OVN Connection

Point Incus to the **northbound** DB (port 6641):

```bash
incus config set lab-node-01: network.ovn.northbound_connection tcp:192.168.102.111:6641
```

### 6.4 Assign OVN Chassis Role

Every node that will host OVN workloads needs the `ovn-chassis` role:

```bash
incus cluster role add lab-node-01:lab-node-01 ovn-chassis
incus cluster role add lab-node-01:lab-node-02 ovn-chassis
incus cluster role add lab-node-01:lab-node-03 ovn-chassis
```

Verify:

```bash
incus cluster list lab-node-01:
```

The ROLES column should now include `ovn-chassis` for each member.

### 6.5 Create UPLINK Physical Network

The UPLINK network provides the bridge between OVN virtual networks and
the physical LAN. It uses the two-step cluster pattern: per-member
`--target` first, then cluster-wide create.

**Per-member configuration** (one per node):

**Important**: IncusOS names its management NIC `mgmt`, NOT `ens18`. Using
`parent=ens18` will fail with "Parent interface 'ens18' not found". Verify
with: `incus query lab-node-01:/os/1.0/system/network`

```bash
incus network create lab-node-01:UPLINK --type physical --target lab-node-01 \
    parent=mgmt

incus network create lab-node-01:UPLINK --type physical --target lab-node-02 \
    parent=mgmt

incus network create lab-node-01:UPLINK --type physical --target lab-node-03 \
    parent=mgmt
```

**Cluster-wide create** with shared settings:

```bash
incus network create lab-node-01:UPLINK --type physical \
    ipv4.ovn.ranges=192.168.103.200-192.168.103.210 \
    ipv4.gateway=192.168.100.1/22 \
    dns.nameservers=192.168.100.1
```

### 6.6 Create OVN Network (net-prod)

```bash
incus network create lab-node-01:net-prod --type=ovn network=UPLINK \
    ipv4.address=10.10.10.1/24 \
    ipv4.nat=true \
    ipv6.address=none
```

Verify:

```bash
incus network list lab-node-01:
```

Should show both `incusbr0` (bridge, per-node) and `net-prod` (ovn, cluster-wide).

### 6.7 Verify Cross-Node OVN Connectivity

Launch containers on different nodes, attached to net-prod:

```bash
incus launch images:debian/12 lab-node-01:test-ovn-a --network net-prod --target lab-node-01
incus launch images:debian/12 lab-node-01:test-ovn-b --network net-prod --target lab-node-02
incus launch images:debian/12 lab-node-01:test-ovn-c --network net-prod --target lab-node-03
```

Wait for IPs and list:

```bash
incus list lab-node-01: --columns nst4 --format csv | grep test-ovn
```

Cross-node ping (node-01 → node-02):

```bash
IP_B=$(incus list lab-node-01:test-ovn-b --columns 4 --format csv | cut -d' ' -f1)
incus exec lab-node-01:test-ovn-a -- ping -c 3 "$IP_B"
```

Actual result: 0% packet loss, ~0.09-0.8ms latency. OVN provides transparent
L2 connectivity via Geneve tunnels.

Cross-node ping (node-01 → node-03):

```bash
IP_C=$(incus list lab-node-01:test-ovn-c --columns 4 --format csv | cut -d' ' -f1)
incus exec lab-node-01:test-ovn-a -- ping -c 3 "$IP_C"
```

Internet access through OVN:

```bash
incus exec lab-node-01:test-ovn-a -- ping -c 3 1.1.1.1
```

Clean up test containers:

```bash
incus delete lab-node-01:test-ovn-a --force
incus delete lab-node-01:test-ovn-b --force
incus delete lab-node-01:test-ovn-c --force
```


## Section 7: Mixed Workloads

Deploy a realistic workload mix: web servers, application containers, and
VMs configured for live migration.

### 7.1 Containers on net-prod

Deploy containers with targeted placement across nodes:

```bash
# Web servers
incus launch images:debian/12 lab-node-01:prod-web-01 --network net-prod --target lab-node-01
incus launch images:debian/12 lab-node-01:prod-web-02 --network net-prod --target lab-node-02

# Application container
incus launch images:debian/12 lab-node-01:prod-api-01 --network net-prod --target lab-node-03
```

Install nginx on the web servers:

```bash
incus exec lab-node-01:prod-web-01 -- bash -c "apt-get update && apt-get install -y nginx"
incus exec lab-node-01:prod-web-02 -- bash -c "apt-get update && apt-get install -y nginx"
```

Set distinct content to verify load balancing later:

```bash
incus exec lab-node-01:prod-web-01 -- bash -c "echo 'Server: prod-web-01' > /var/www/html/index.html"
incus exec lab-node-01:prod-web-02 -- bash -c "echo 'Server: prod-web-02' > /var/www/html/index.html"
```

Install nginx on the API container:

```bash
incus exec lab-node-01:prod-api-01 -- bash -c "apt-get update && apt-get install -y nginx"
incus exec lab-node-01:prod-api-01 -- bash -c "echo 'API: prod-api-01' > /var/www/html/index.html"
```

### 7.2 VMs (Migration-Ready)

Deploy VMs with live migration configuration:

```bash
incus launch images:debian/12 lab-node-01:prod-db-01 --vm --network net-prod --target lab-node-01
incus launch images:debian/12 lab-node-01:prod-app-01 --vm --network net-prod --target lab-node-02
```

VMs may take longer to boot than containers (~30-60s for image download +
boot). If the VMs show as STOPPED, start them explicitly:

```bash
incus start lab-node-01:prod-db-01
incus start lab-node-01:prod-app-01
```

Wait for the VM agent to become available, then verify:

```bash
# Check VM agent is running
incus exec lab-node-01:prod-db-01 -- uname -a
incus exec lab-node-01:prod-app-01 -- uname -a
```

### 7.3 Configure VMs for Live Migration

**Critical**: use `limits.cpu` as a **range** (e.g., `0-1`), not an integer.
Without the range, QEMU sets `maxcpus` based on the host's CPU count, which
varies across nodes and breaks migration with `Missing section footer for
ICH9LPC`.

Stop VMs before configuring `migration.stateful`:

```bash
incus stop lab-node-01:prod-db-01
incus stop lab-node-01:prod-app-01
```

Configure migration settings:

```bash
# prod-db-01
incus config set lab-node-01:prod-db-01 limits.cpu=0-1
incus config set lab-node-01:prod-db-01 migration.stateful=true
incus config device set lab-node-01:prod-db-01 root size.state=2GiB

# prod-app-01
incus config set lab-node-01:prod-app-01 limits.cpu=0-1
incus config set lab-node-01:prod-app-01 migration.stateful=true
incus config device set lab-node-01:prod-app-01 root size.state=2GiB
```

Start the VMs:

```bash
incus start lab-node-01:prod-db-01
incus start lab-node-01:prod-app-01
```

### 7.4 Workload Distribution

View the full workload distribution:

```bash
incus list lab-node-01: --columns nstL4 --format table
```

Expected layout:

```
+--------------+---------+-------------------+-------------+-----------------------+
|     NAME     |  STATE  |       TYPE        |  LOCATION   |         IPV4          |
+--------------+---------+-------------------+-------------+-----------------------+
| ovn-central  | RUNNING | CONTAINER         | lab-node-01 | ...                   |
| prod-web-01  | RUNNING | CONTAINER         | lab-node-01 | 10.10.10.x (net-prod) |
| prod-db-01   | RUNNING | VIRTUAL-MACHINE   | lab-node-01 | 10.10.10.x (net-prod) |
| prod-web-02  | RUNNING | CONTAINER         | lab-node-02 | 10.10.10.x (net-prod) |
| prod-app-01  | RUNNING | VIRTUAL-MACHINE   | lab-node-02 | 10.10.10.x (net-prod) |
| prod-api-01  | RUNNING | CONTAINER         | lab-node-03 | 10.10.10.x (net-prod) |
+--------------+---------+-------------------+-------------+-----------------------+
```


## Section 8: Network Isolation & Security

### 8.1 Create Isolated Network

```bash
incus network create lab-node-01:net-isolated --type=ovn network=UPLINK \
    ipv4.address=10.10.20.1/24 \
    ipv4.nat=true \
    ipv6.address=none
```

### 8.2 Launch Isolated Containers

```bash
incus launch images:debian/12 lab-node-01:iso-app-01 --network net-isolated --target lab-node-01
incus launch images:debian/12 lab-node-01:iso-app-02 --network net-isolated --target lab-node-02
```

### 8.3 Verify Network Isolation

Containers on net-isolated can reach each other:

```bash
IP_ISO2=$(incus list lab-node-01:iso-app-02 --columns 4 --format csv | cut -d' ' -f1)
incus exec lab-node-01:iso-app-01 -- ping -c 3 "$IP_ISO2"
```

Actual result: 0% packet loss, ~0.15-0.5ms latency. Containers on the
same OVN network can reach each other across nodes.

But net-prod **cannot** reach net-isolated:

```bash
incus exec lab-node-01:prod-web-01 -- ping -c 3 -W 2 "$IP_ISO2"
```

Actual result: **100% packet loss**. Different OVN networks are fully
isolated — separate L2 domains, no routing between them.

### 8.4 Create Network ACL

Create an ACL that blocks ICMP from a specific source:

```bash
incus network acl create lab-node-01:block-ping
incus network acl rule add lab-node-01:block-ping ingress \
    action=drop protocol=icmp4 \
    source=10.10.10.0/24 \
    description="Block ICMP from net-prod subnet"
```

### 8.5 Apply and Test ACL

Apply the ACL to net-isolated:

```bash
incus network set lab-node-01:net-isolated security.acls=block-ping
```

Verify ICMP is blocked between net-isolated containers (since they match
the source range — adjust the ACL source for targeted blocking):

```bash
incus exec lab-node-01:iso-app-01 -- ping -c 3 -W 2 "$IP_ISO2"
```

Remove the ACL:

```bash
incus network unset lab-node-01:net-isolated security.acls
```

Verify ICMP works again:

```bash
incus exec lab-node-01:iso-app-01 -- ping -c 3 "$IP_ISO2"
```

### 8.6 Network Peering

Connect net-prod and net-isolated so containers on both networks can
communicate. Peering is bilateral — create a peer on both sides:

```bash
# From net-prod's perspective
incus network peer create lab-node-01:net-prod peer-to-isolated net-isolated \
    --description "Peer to isolated network"

# From net-isolated's perspective
incus network peer create lab-node-01:net-isolated peer-to-prod net-prod \
    --description "Peer to production network"
```

### 8.7 Verify Peering

Cross-network ping (prod → isolated):

```bash
incus exec lab-node-01:prod-web-01 -- ping -c 3 "$IP_ISO2"
```

Actual result: 0% packet loss with TTL=62 (64 - 2 router hops),
confirming traffic traverses the OVN routers on both sides of the peering.

Cross-network ping (isolated → prod):

```bash
IP_WEB1=$(incus list lab-node-01:prod-web-01 --columns 4 --format csv | cut -d' ' -f1)
incus exec lab-node-01:iso-app-01 -- ping -c 3 "$IP_WEB1"
```

### 8.8 Remove Peering

```bash
incus network peer delete lab-node-01:net-prod peer-to-isolated
incus network peer delete lab-node-01:net-isolated peer-to-prod
```

Verify isolation is restored:

```bash
incus exec lab-node-01:prod-web-01 -- ping -c 3 -W 2 "$IP_ISO2"
```

Expected: 100% packet loss. Networks are isolated again.

Clean up isolated containers:

```bash
incus delete lab-node-01:iso-app-01 --force
incus delete lab-node-01:iso-app-02 --force
```


## Section 9: Load Balancers & Network Forwards

### 9.1 Create OVN Load Balancer

Create a load balancer with a VIP from the UPLINK range:

```bash
incus network load-balancer create lab-node-01:net-prod 192.168.103.200
```

Add backend servers. **Important**: backends require the instance's **IP
address**, not its name. Get the IPs first:

```bash
WEB1_IP=$(incus list lab-node-01:prod-web-01 --columns 4 --format csv | cut -d' ' -f1)
WEB2_IP=$(incus list lab-node-01:prod-web-02 --columns 4 --format csv | cut -d' ' -f1)
echo "prod-web-01: $WEB1_IP, prod-web-02: $WEB2_IP"
```

Add backends using IP addresses:

```bash
incus network load-balancer backend add lab-node-01:net-prod 192.168.103.200 \
    web-01 "$WEB1_IP" 80
incus network load-balancer backend add lab-node-01:net-prod 192.168.103.200 \
    web-02 "$WEB2_IP" 80
```

Add a port mapping:

```bash
incus network load-balancer port add lab-node-01:net-prod 192.168.103.200 \
    tcp 80 web-01,web-02
```

### 9.2 Test Load Balancer

From your dev machine (must be on the same VLAN or have routing to
192.168.103.0/24):

```bash
for i in $(seq 1 6); do
    curl -s http://192.168.103.200
done
```

Actual output:

```
Server: prod-web-01
Server: prod-web-01
Server: prod-web-01
Server: prod-web-02
Server: prod-web-02
Server: prod-web-02
```

OVN uses connection-based hashing (not round-robin). Multiple requests from
the same source will typically hit the same backend. Different source ports
or connections may hit different backends.

### 9.3 Create Network Forward

Network forwards expose internal services on LAN IPs. Forward
tcp:8080 → prod-api-01:80. **Like LB backends, forwards require IP addresses**:

```bash
API_IP=$(incus list lab-node-01:prod-api-01 --columns 4 --format csv | cut -d' ' -f1)

incus network forward create lab-node-01:net-prod 192.168.103.201
incus network forward port add lab-node-01:net-prod 192.168.103.201 \
    tcp 8080 "$API_IP" 80
```

### 9.4 Test Network Forward

```bash
curl -s http://192.168.103.201:8080
```

Actual output: `API: prod-api-01`

### 9.5 DNS Resolution

OVN provides per-network DNS. Containers can resolve each other by
hostname:

```bash
incus exec lab-node-01:prod-web-01 -- bash -c "apt-get install -y dnsutils && dig +short prod-web-02.incus"
```

Actual output: `10.10.10.3` — OVN DNS resolves instance names within
each network.


## Section 10: Live Migration

### 10.1 Verify Migration Readiness

Check that VMs have the required configuration:

```bash
for vm in prod-db-01 prod-app-01; do
    echo "=== $vm ==="
    incus config get lab-node-01:$vm limits.cpu
    incus config get lab-node-01:$vm migration.stateful
    incus config device get lab-node-01:$vm root size.state
done
```

Expected: `0-1`, `true`, `2GiB` for each VM.

### 10.2 Create Heartbeat Service

Create a simple counter in prod-db-01 to verify state continuity across
migration:

```bash
incus exec lab-node-01:prod-db-01 -- bash -c '
    mkdir -p /tmp/heartbeat
    nohup bash -c "i=0; while true; do echo \$i > /tmp/heartbeat/counter; i=\$((i+1)); sleep 1; done" \
        > /dev/null 2>&1 &
    echo "Heartbeat started"
'
```

Read the counter:

```bash
incus exec lab-node-01:prod-db-01 -- cat /tmp/heartbeat/counter
```

Note the value. After migration, the counter should continue from where
it left off (live migration preserves running state).

### 10.3 Live Migration Round-Trip

Check current location:

```bash
incus list lab-node-01:prod-db-01 --columns nL --format csv
```

**Migrate node-01 → node-02:**

```bash
time incus move lab-node-01:prod-db-01 --target lab-node-02
```

Actual result: 7.347s (~140 MB/s). Wait for the VM agent to reconnect:

```bash
sleep 4
incus exec lab-node-01:prod-db-01 -- cat /tmp/heartbeat/counter
```

Counter went from 9 → 25. The heartbeat process was never interrupted —
it continued counting during migration.

**Migrate node-02 → node-03:**

```bash
time incus move lab-node-01:prod-db-01 --target lab-node-03
sleep 4
incus exec lab-node-01:prod-db-01 -- cat /tmp/heartbeat/counter
```

Actual result: 7.379s. Counter went to 41.

**Migrate node-03 → node-01 (back to origin):**

```bash
time incus move lab-node-01:prod-db-01 --target lab-node-01
sleep 4
incus exec lab-node-01:prod-db-01 -- cat /tmp/heartbeat/counter
```

Actual result: 6.896s. Counter went to 56.

Verify the VM is back on node-01:

```bash
incus list lab-node-01:prod-db-01 --columns nL --format csv
```

### 10.4 Active I/O During Migration

Start a continuous write inside the VM:

```bash
incus exec lab-node-01:prod-db-01 -- bash -c '
    dd if=/dev/urandom of=/tmp/testfile bs=1M count=100 &
    echo "Write started, PID: $!"
'
```

Migrate while I/O is active:

```bash
time incus move lab-node-01:prod-db-01 --target lab-node-02
sleep 4
```

Verify the file exists and is intact:

```bash
incus exec lab-node-01:prod-db-01 -- ls -la /tmp/testfile
incus exec lab-node-01:prod-db-01 -- md5sum /tmp/testfile
```

Move back:

```bash
incus move lab-node-01:prod-db-01 --target lab-node-01
sleep 4
```

### 10.5 Stateful Stop/Restore

Stateful stop saves VM memory to disk. On start, the VM resumes exactly
where it was:

```bash
# Note the heartbeat counter
incus exec lab-node-01:prod-app-01 -- bash -c '
    mkdir -p /tmp/heartbeat
    echo 42 > /tmp/heartbeat/counter
    cat /tmp/heartbeat/counter
'
```

Stateful stop:

```bash
incus stop lab-node-01:prod-app-01 --stateful
```

Start (resumes from saved state):

```bash
incus start lab-node-01:prod-app-01
sleep 4
incus exec lab-node-01:prod-app-01 -- cat /tmp/heartbeat/counter
```

Expected: `42` — the file (and entire VM state) is preserved.

**If the restore fails** (e.g., from a `limits.cpu` mismatch), discard the
saved state:

```bash
incus start lab-node-01:prod-app-01 --stateless
```


## Section 11: Cluster Lifecycle

### 11.1 Evacuation & Restore

Evacuate node-02. All workloads are moved to other nodes:

```bash
incus cluster evacuate lab-node-01:lab-node-02 --force
```

Check workload distribution — nothing on node-02:

```bash
incus list lab-node-01: --columns nstL --format table
```

Actual behavior: VMs with `migration.stateful=true` are live-migrated
(prod-app-01 migrated to lab-node-03). Containers are stopped and moved
(prod-web-02 stopped, moved to lab-node-03, then started). The `--force`
flag skips confirmation prompts.

**Note**: if VMs lack the `limits.cpu` range fix, use `--action stop`
instead to avoid migration failures:
```bash
incus cluster evacuate lab-node-01:lab-node-02 --force --action stop
```

Verify node-02 shows EVACUATED:

```bash
incus cluster list lab-node-01:
```

Restore node-02 (workloads return):

```bash
incus cluster restore lab-node-01:lab-node-02 --force
```

Verify all workloads are back:

```bash
incus list lab-node-01: --columns nstL --format table
incus cluster list lab-node-01:
```

All nodes should show ONLINE.

### 11.2 Node Failure Simulation

A Proxmox hard-stop on a VM simulates a crash. The Incus cluster heartbeat
detects the failure in ~40 seconds. After the node is restarted:

1. The node auto-rejoins the cluster (~60s)
2. Containers auto-start
3. VMs that were running resume

**Procedure** (document only — do not execute while OVN is running unless
you can tolerate temporary network disruption):

```bash
# Simulate crash: hard-stop via Proxmox API
# curl -s -k -X POST "https://192.168.1.29:8006/api2/json/nodes/pve/qemu/912/status/stop" ...

# Wait for heartbeat detection (~40s)
# incus cluster list lab-node-01:
# → lab-node-02 shows OFFLINE

# Restart via Proxmox
# curl -s -k -X POST "https://192.168.1.29:8006/api2/json/nodes/pve/qemu/912/status/start" ...

# Wait for auto-rejoin (~60s)
# incus cluster list lab-node-01:
# → lab-node-02 shows ONLINE
```

### 11.3 Node Replacement

Full procedure: evacuate a node, remove it from the cluster, destroy the
VM, deploy a fresh node, and join it back. This tests the complete lifecycle.

**Step 1: Evacuate node-03:**

```bash
incus cluster evacuate lab-node-01:lab-node-03 --force --action stop
```

**Step 2: Remove from cluster:**

```bash
printf "yes\n" | incus cluster remove lab-node-01:lab-node-03 --force
```

**Note**: `incus cluster remove` prompts "Are you really sure?" even with
`--force`. The `printf` pipes `yes` for automation.

**Step 3: Clean up the remote:**

```bash
incus remote remove lab-node-03
```

**Step 4: Destroy and redeploy the VM.** Use `incusos-proxmox` to destroy
just node-03 and redeploy it. The simplest approach: create a single-VM
config or use the replacement config pattern:

```bash
# Destroy just node-03 via Proxmox API (VMID 913)
# Then redeploy with incusos-proxmox using a config that only defines node-03
```

Alternatively, if you have a `lab-replace.yaml` config for the replacement
node:

```bash
./incusos-proxmox --yes examples/lab-replace.yaml
```

**Step 5: Join the fresh node to the cluster:**

```bash
# Set specific IP
incus config set lab-node-03: core.https_address 192.168.102.113:8443

# Generate join token
incus cluster add lab-node-01:lab-node-03

# Join
printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join lab-node-01: lab-node-03:

# Fix remote
incus remote remove lab-node-03
incus remote add lab-node-03 https://192.168.102.113:8443 --accept-certificate
```

**Step 6: Re-enable OVN on the replacement node:**

```bash
incus query lab-node-03:/os/1.0/services/ovn --request PUT --data '{
  "config": {
    "database": "tcp:192.168.102.111:6642",
    "enabled": true,
    "tunnel_address": "192.168.102.113",
    "tunnel_protocol": "geneve"
  },
  "state": {}
}'

incus cluster role add lab-node-01:lab-node-03 ovn-chassis
```

**Step 7: Verify:**

```bash
incus cluster list lab-node-01:
```

All 3 nodes should be ONLINE with `ovn-chassis` role.

### 11.4 Cluster Rebalancing

Enable automatic workload rebalancing. When a new node joins (or workloads
are unevenly distributed), Incus redistributes VMs:

```bash
incus config set lab-node-01: cluster.rebalance.interval=1
incus config set lab-node-01: cluster.rebalance.threshold=10
incus config set lab-node-01: cluster.rebalance.batch=2
incus config set lab-node-01: cluster.rebalance.cooldown=5m
```

**Important**: only VMs with `migration.stateful=true` are rebalanced.
Containers are NOT auto-rebalanced.

Monitor rebalancing:

```bash
incus list lab-node-01: --columns nstL --format table
```

Disable rebalancing when done testing:

```bash
incus config unset lab-node-01: cluster.rebalance.interval
incus config unset lab-node-01: cluster.rebalance.threshold
```


## Section 12: OC Dashboard

**Important**: Switch to the OC remote first. The OC CLI does not support
`remote:` suffix syntax:

```bash
operations-center remote switch oc-lab
```

### OC Server Information

```bash
operations-center admin os show
```

Actual output:

```
+-------------------+----------------------------+
|     PROPERTY      |           VALUE            |
+-------------------+----------------------------+
| hostname          | lab-oc                     |
| os_version        | 202602210344               |
| kernel            | 6.12.13                    |
| architecture      | x86_64                     |
| uptime            | 4390                       |
| addresses         | 192.168.102.110/22 (mgmt)  |
|                   | fd42:...:1 (incusbr0)      |
| storage_disks     | /dev/sda (53.7GB, QEMU)    |
| storage_pools     | local (zfs, /dev/sda4)     |
+-------------------+----------------------------+
```

### Application Status

```bash
operations-center admin os application list
```

Actual output:

```
+--------------------+---------+
|        NAME        | STATUS  |
+--------------------+---------+
| operations-center  | running |
+--------------------+---------+
```

### Service Status

```bash
operations-center admin os service list
```

Actual output:

```
+----------+---------+
|   NAME   | ENABLED |
+----------+---------+
| ovn      | false   |
| syslog   | false   |
| fan      | false   |
| bgp      | false   |
| dns      | false   |
| metricsA | false   |
| metricsB | false   |
+----------+---------+
```

### Web UI

The OC web UI at `https://192.168.102.110:8443/ui/` provides:

- **Dashboard**: server overview with resource utilization
- **Updates**: available IncusOS updates
- **Provisioning**: token management (for OC-provisioned deployments)
- **System**: OC configuration and certificates

**Limitation**: OC deployed with a standard ISO cannot manage the Incus
cluster nodes. The nodes are independent — they were deployed with a
standard IncusOS ISO, not an OC-provisioned one. For full OC node
management (cluster orchestration, application deployment, monitoring),
nodes must boot from an OC-provisioned ISO.

See [Operations Center Guide](operations-center-guide.md) for the full
hybrid deployment workflow with OC-provisioned ISOs.


## Section 13: Cleanup

### Delete All Workloads

```bash
# Delete containers
for c in prod-web-01 prod-web-02 prod-api-01; do
    incus delete lab-node-01:$c --force
done

# Delete VMs
for vm in prod-db-01 prod-app-01; do
    incus delete lab-node-01:$vm --force
done
```

### Remove OVN Networks

```bash
# Delete OVN networks
incus network delete lab-node-01:net-prod
incus network delete lab-node-01:net-isolated 2>/dev/null || true

# Delete UPLINK
incus network delete lab-node-01:UPLINK
```

### Remove OVN Control Plane

```bash
incus delete lab-node-01:ovn-central --force
```

### Disable OVN Services

```bash
for node in lab-node-01 lab-node-02 lab-node-03; do
    incus query "$node":/os/1.0/services/ovn --request PUT --data '{
      "config": {
        "enabled": false
      },
      "state": {}
    }'
done
```

### Infrastructure Options

**Keep infrastructure** (stop VMs, keep on disk for later):

```bash
./incusos-proxmox --lab-down examples/lab-production.yaml
```

Restart later with:

```bash
./incusos-proxmox --lab-up examples/lab-production.yaml
```

**Full teardown** (destroy all VMs, remove ISOs, remotes, cache):

```bash
./incusos-proxmox --cleanup --deep examples/lab-production.yaml
```


## Section 14: Verification Checklist

| # | Check | Command | Expected |
|---|-------|---------|----------|
| 1 | All VMs running | `incusos-proxmox --status examples/lab-production.yaml` | 4 VMs running, port 8443 open |
| 2 | Scrub schedule healthy | `incus query lab-node-01:/os/1.0/system/storage` | `scrub_schedule: "0 4 * * 0"` |
| 3 | OC accessible | `operations-center remote switch oc-lab && operations-center admin os show` | Shows version, uptime |
| 4 | Cluster formed | `incus cluster list lab-node-01:` | 3 nodes ONLINE |
| 5 | Storage pool | `incus storage list lab-node-01:` | `local` pool on all members |
| 6 | Bridge isolation | Ping cross-node on incusbr0 | 100% loss (expected) |
| 7 | OVN connectivity | Ping cross-node on net-prod | 0% loss |
| 8 | Internet via OVN | `ping 1.1.1.1` from OVN container | 0% loss |
| 9 | Network isolation | Ping net-prod → net-isolated | 100% loss (expected) |
| 10 | Network peering | Peer + ping cross-network | 0% loss, TTL=62 |
| 11 | Load balancer | `curl http://192.168.103.200` | Backend response |
| 12 | Network forward | `curl http://192.168.103.201:8080` | API response |
| 13 | DNS resolution | `dig prod-web-02.incus` from container | Resolves to 10.10.10.x |
| 14 | VM live migration | `incus move` VM between nodes | State preserved |
| 15 | Cluster evacuation | `incus cluster evacuate` + `restore` | Workloads moved and returned |
| 16 | Stateful stop/start | `incus stop --stateful` + `start` | VM state preserved |


## Section 15: Quick Reference

### Cluster Command Syntax

| Command | Arguments | Notes |
|---------|-----------|-------|
| `incus cluster enable` | `remote: member-name` | TWO args (space between) |
| `incus cluster add` | `remote:member-name` | ONE arg (no space) |
| `incus cluster join` | `init-remote: joining-remote:` | TWO args (space between) |
| `incus cluster remove` | `remote:member-name --force` | ONE arg; prompts even with `--force` |
| `incus cluster evacuate` | `remote:member-name` | ONE arg (no space) |
| `incus cluster restore` | `remote:member-name` | ONE arg (no space) |
| `incus config set` | `remote: key value` | Remote with trailing colon + space |
| `incus storage show` | `remote:pool` | ONE arg (no space) |
| `incus storage show` | `remote:pool --target member` | `--target` for member-specific |

### OVN Setup Cheat Sheet

```bash
# 1. Deploy OVN container
incus launch images:debian/12 REMOTE:ovn-central --target NODE
incus exec REMOTE:ovn-central -- apt-get install -y ovn-central ovn-host
incus exec REMOTE:ovn-central -- ovn-nbctl set-connection ptcp:6641:0.0.0.0
incus exec REMOTE:ovn-central -- ovn-sbctl set-connection ptcp:6642:0.0.0.0
incus config device add REMOTE:ovn-central nb-proxy proxy listen=tcp:HOST_IP:6641 connect=tcp:127.0.0.1:6641
incus config device add REMOTE:ovn-central sb-proxy proxy listen=tcp:HOST_IP:6642 connect=tcp:127.0.0.1:6642

# 2. Enable OVN on each IncusOS node
incus query NODE:/os/1.0/services/ovn --request PUT --data '{"config":{"database":"tcp:HOST_IP:6642","enabled":true,"tunnel_address":"NODE_IP","tunnel_protocol":"geneve"},"state":{}}'

# 3. Configure Incus
incus config set REMOTE: network.ovn.northbound_connection tcp:HOST_IP:6641
incus cluster role add REMOTE:MEMBER ovn-chassis  # for each member

# 4. Create UPLINK (per-member then cluster-wide)
incus network create REMOTE:UPLINK --type physical --target MEMBER parent=mgmt  # each member
incus network create REMOTE:UPLINK --type physical ipv4.ovn.ranges=RANGE ipv4.gateway=GW/PREFIX

# 5. Create OVN network
incus network create REMOTE:net-name --type=ovn network=UPLINK ipv4.address=SUBNET ipv4.nat=true
```

### Migration Readiness Checklist

| Setting | Value | Why |
|---------|-------|-----|
| `limits.cpu` | Range (e.g., `0-1`) | Fixed QEMU topology across hosts |
| `migration.stateful` | `true` | Enables live migration |
| `root size.state` | `2GiB` (or `4GiB` for 3-4 vCPUs) | Space for memory state file |

Configure while VM is **stopped**:

```bash
incus stop REMOTE:VM
incus config set REMOTE:VM limits.cpu=0-1
incus config set REMOTE:VM migration.stateful=true
incus config device set REMOTE:VM root size.state=2GiB
incus start REMOTE:VM
```

### Troubleshooting

| Symptom | Cause | Fix |
|---------|-------|-----|
| Port 8443 not reachable after boot | Boot still in progress or crontab bug | Wait 180s; check scrub_schedule via API |
| `scrub_schedule` empty | Crontab race condition | `incusos-proxmox --status` auto-heals |
| `Missing section footer for ICH9LPC` on migration | `limits.cpu` set as integer | Set as range: `limits.cpu=0-1` |
| `VM agent isn't currently running` after migration | Agent reconnecting | `sleep 4` after migration |
| `db.sock not found` on OVN config | OVN service not enabled on IncusOS | Enable via `/os/1.0/services/ovn` API |
| Cross-node ping fails (bridge) | Bridge networks are node-local | Use OVN network instead |
| `zfs load-key: Raw key too short` | TPM corruption from premature VM stop | Destroy and redeploy VM |
| Cluster join fails with "pool already exists" | `apply_defaults: true` on joining node | Use `apply_defaults: false` or run 8-command cleanup |
| OC cannot manage cluster nodes | Nodes deployed with standard ISO | Use OC-provisioned ISO for full integration |
| `CPUID vnmi` warning during migration | Cosmetic QEMU check | Safe to ignore |
| "Parent interface 'ens18' not found" | IncusOS names its NIC `mgmt` | Use `parent=mgmt` for UPLINK network |
| "Invalid target address" on LB backend | Backend needs IP, not instance name | Use instance IP address (e.g., `10.10.10.2`) |
| OC CLI "Invalid number of arguments" | OC CLI doesn't support `remote:` suffix | Use `operations-center remote switch NAME` first |
| Container placed on wrong node | Cluster auto-schedules without `--target` | Use `--target NODE` for explicit placement |