# Production Home Lab Guide

Build a production-quality Incus home lab from scratch: Operations Center
dashboard, OC-managed 3-node cluster with OVN overlay networking, Aether
management platform, HAProxy load balancing, AWX lifecycle automation,
Prometheus/Grafana/Loki observability stack, live migration, network
security, and cluster lifecycle management.

All commands and output in this guide are from actual deployments on
Proxmox VE 9.1.5 with IncusOS 202602230420, Incus client 6.21,
Operations Center v0.3.0, and Aether v6.4.317.


## Section 0: Architecture Overview

### Network Topology

```mermaid
flowchart TD
    vlan(("VLAN 69<br/>192.168.100.0/22"))

    subgraph mgmt["Management"]
        oc["oc-server<br/>VMID 920 · .120<br/>Operations Center"]
    end

    subgraph cluster["Incus Cluster"]
        n1["oc-node-01<br/>VMID 400 · .140<br/>Aether · OVN central"]
        n2["oc-node-02<br/>VMID 401 · .141<br/>AWX · Monitoring"]
        n3["oc-node-03<br/>VMID 402 · .142<br/>HAProxy backends"]
    end

    subgraph services["Macvlan Services"]
        aether["Aether · .160"]
        awx["AWX · .161"]
    end

    subgraph ovn["OVN · net-prod 10.10.10.0/24"]
        haproxy["HAProxy HA<br/>.50 · .51"]
        backends["nginx backends<br/>.60 · .61 · .62"]
        monitoring["Observability<br/>.70 + node-exp .71-.73"]
    end

    subgraph uplink["UPLINK · 192.168.103.x"]
        vip[".200 HAProxy VIP"]
        fwd[".201 Grafana / Prometheus"]
    end

    vlan --- mgmt & cluster
    cluster --> services & ovn
    ovn -.-> uplink

    classDef nodeClass fill:#009E73,color:#fff,stroke:#007a5e
    classDef mgmtClass fill:#CC79A7,color:#fff,stroke:#a36088
    classDef serviceClass fill:#E69F00,color:#fff,stroke:#b87d00
    classDef ovnClass fill:#56B4E9,color:#fff,stroke:#3a8fbf
    classDef networkClass fill:#0072B2,color:#fff,stroke:#005a8e

    class n1,n2,n3 nodeClass
    class oc mgmtClass
    class aether,awx serviceClass
    class haproxy,backends,monitoring ovnClass
    class vip,fwd,vlan networkClass

    style mgmt fill:#f5e6f0,stroke:#CC79A7
    style cluster fill:#e6f5f0,stroke:#009E73
    style services fill:#fef3e0,stroke:#E69F00
    style ovn fill:#e0f2fe,stroke:#56B4E9
    style uplink fill:#e0eef8,stroke:#0072B2
```

### Infrastructure

| Component | VMID | IP | Cores | RAM | Disk | Role |
|-----------|------|-----|-------|-----|------|------|
| oc-server | 920 | 192.168.102.120/22 | 2 | 4 GiB | 50G | Operations Center |
| oc-node-01 | 400 | 192.168.102.140/22 | 4 | 20 GiB | 250G | Cluster init + Aether host |
| oc-node-02 | 401 | 192.168.102.141/22 | 4 | 20 GiB | 100G | AWX + monitoring host |
| oc-node-03 | 402 | 192.168.102.142/22 | 4 | 20 GiB | 100G | HAProxy backends |

**RAM budget**: 64 GiB of 94 GiB (68%). Host: i9-13900HK, 94 GiB RAM,
881 GiB ZFS pool. Leaves 30 GiB headroom for other VMs on the host.

### Inner Cluster Services

| Instance | Network | IP | Node | RAM | Description |
|----------|---------|-----|------|-----|-------------|
| ovn-central | incusbr0 | DHCP | node-01 | 512 MiB | OVN NB/SB databases |
| aether | macvlan mgmt | 192.168.102.160 | node-01 | 8 GiB | Management platform |
| awx | macvlan mgmt | 192.168.102.161 | node-02 | 8 GiB | Ansible automation |
| ha-web-01 | net-prod | 10.10.10.60 | node-01 | 256 MiB | Nginx backend |
| ha-web-02 | net-prod | 10.10.10.61 | node-02 | 256 MiB | Nginx backend |
| ha-web-03 | net-prod | 10.10.10.62 | node-03 | 256 MiB | Nginx backend |
| haproxy-01 | net-prod | 10.10.10.50 | varies | 512 MiB | HA load balancer |
| haproxy-02 | net-prod | 10.10.10.51 | varies | 512 MiB | HA load balancer |
| monitoring | net-prod | 10.10.10.70 | node-02 | 2 GiB | Prometheus + Grafana + Loki |
| node-exp-01 | net-prod | 10.10.10.71 | node-01 | 128 MiB | Host metrics exporter |
| node-exp-02 | net-prod | 10.10.10.72 | node-02 | 128 MiB | Host metrics exporter |
| node-exp-03 | net-prod | 10.10.10.73 | node-03 | 128 MiB | Host metrics exporter |

### External IP Allocation

OVN external addresses from UPLINK range (192.168.103.200-210). Exclude
these from your DHCP server's range:

| IP | Purpose |
|-----|---------|
| 192.168.103.200 | HAProxy VIP (OVN load balancer → haproxy-01/02) |
| 192.168.103.201 | Observability forward (Grafana :3000, Prometheus :9090) |

### Decision Rationale

**Why OC-managed clustering?** OC `provisioning cluster add` is the
production path for Incus deployments. It handles cluster formation,
update management, and inventory centrally. The deploy scripts
(`deploy-haproxy`, `deploy-awx`, `deploy-observability`) are built
for the `oc-node-*` naming and IP scheme.

**Why 20 GiB RAM per node?** Aether requires 8 GiB. AWX requires
4-8 GiB. Monitoring + HAProxy + backends need ~3 GiB total.
Leaves headroom for mixed workloads and live migration.

**Why 250 GiB disk for node-01?** Aether's golden image is 200 GiB
virtual (qcow2). With ZFS thin provisioning only ~11 GiB is used
initially, but the pool needs 200 GiB allocatable space.

**Why OVN?** Bridge networks are node-local — instances on different
nodes cannot communicate. OVN provides cross-node L2 overlay with
sub-ms latency, network isolation, ACLs, load balancers, and network
forwards — essential for HAProxy HA and distributed workloads.

**Why VLAN 69?** Isolates lab traffic from the production LAN. All VMs
share VLAN 69 (subnet 192.168.100.0/22). The VLAN tag is set at the
Proxmox level — IncusOS and workloads are unaware of it.

### Cross-References

This guide integrates techniques from the deep-dive guides:

- [Clustering Guide](clustering-guide.md) — manual cluster formation reference
- [Networking Guide](networking-guide.md) — OVN setup, ACLs, peering
- [Operations Center Guide](operations-center-guide.md) — OC provisioning, CLI, web UI
- [Aether Guide](aether-guide.md) — management platform deployment and API
- [HAProxy Guide](haproxy-guide.md) — HA load balancing with Aether
- [AWX Guide](awx-guide.md) — Ansible lifecycle automation
- [Observability Guide](observability-guide.md) — Prometheus, Grafana, Loki stack


## Section 1: Prerequisites

### Required Tools

Verify all tools are available before starting:

```bash
incus version
operations-center --version
bash --version | head -1
python3 --version
jq --version
curl --version | head -1
genisoimage --version 2>&1 | head -1
```

**Minimum versions**: Incus client 6.3+, Operations Center v0.3.0+.

### Aether Browser Automation

Several Aether features (HAProxy management, blueprint deployment) are
not in the JWT API — they use session-authenticated routes with CSRF
protection. Playwright browser automation is required for Sections 7-9:

```bash
node --version         # Node.js 18+
npx playwright --version
```

Install if missing:

```bash
npm install playwright @playwright/mcp
npx playwright install chromium
```

The Playwright MCP server (configured in `.mcp.json`) provides browser
tools when available. The `incusos/helpers/aether-browser` script is
the standalone alternative.

### Aether Golden Image

The Aether golden image must be available locally before Section 7:

```bash
ls -la sources/aether-golden-image-v6.tar.gz
```

This is an Ubuntu Noble (24.04) image with 200 GiB virtual disk (~6.6
GiB compressed). Obtain it from the Aether distribution.

### Proxmox Configuration

Your `incusos/proxmox.yaml` should contain:

```yaml
host: 192.168.1.29
method: api
api_token_id: automation@pve!deploy
node: pve
storage: local-zfs
iso_storage: local
bridge: vmbr0
vlan: 69
gateway: 192.168.100.1
dns: 192.168.100.1
pool: IncusLab
```

The `env` file at the repository root must export `PROXMOX_TOKEN_SECRET`
and `AETHER_ADMIN_PASSWORD`. Scripts auto-discover them.

### Client Certificates

Incus client certificates are used for Incus, OC, and Prometheus
(metrics scraping) connections:

```bash
# Verify cert exists (auto-generated on first incus command)
ls -la ~/.config/incus/client.crt ~/.config/incus/client.key
```

Copy certs for OC CLI:

```bash
mkdir -p ~/.config/operations-center
cp ~/.config/incus/client.crt ~/.config/operations-center/
cp ~/.config/incus/client.key ~/.config/operations-center/
```

For OC web UI browser access, generate a PKCS#12 bundle:

```bash
openssl pkcs12 -export \
    -out ~/.config/incus/client.pfx \
    -inkey ~/.config/incus/client.key \
    -in ~/.config/incus/client.crt \
    -name "Incus Client"
```

Import `client.pfx` into your browser's certificate store (Firefox:
Settings → Privacy & Security → View Certificates → Import).

### Doctor Check

Run the environment check to verify everything is in order:

```bash
cd incusos
./incusos-proxmox --doctor
```

Expected output includes tool versions, IncusOS CDN reachability,
`proxmox.yaml` discovery, and Proxmox API connectivity.


## Section 2: Deploy OC Server

### Configuration File

```yaml
# incusos/examples/lab-oc-deploy.yaml
defaults:
  cores: 2
  memory: 4096
  disk: 50
  start_vmid: 920

proxmox:
  gateway: 192.168.100.1
  dns: 192.168.100.1

vms:
  - name: oc-server
    app: operations-center
    apply_defaults: true
    ip: 192.168.102.120/22
```

### Deploy

```bash
./incusos-proxmox --yes incusos/examples/lab-oc-deploy.yaml
```

Actual output (key lines):

```
[ok]  VM 'oc-server' created (VMID 920)
[ok]  VM 'oc-server' installed and running at 192.168.102.120
```

### Set Up OC CLI Remote

```bash
# Accept the TLS certificate when prompted
operations-center remote add oc-lab https://192.168.102.120:8443 --auth-type tls
operations-center remote switch oc-lab
```

**Important**: The OC CLI does **not** support the `remote:` suffix syntax
that the Incus CLI uses. Switch to the remote first, then run commands
without a remote suffix.

### Verify OC

```bash
operations-center admin os show
```

Actual output (uptime will vary):

```
WARNING: The IncusOS API and configuration is subject to change

environment:
  hostname: oc-server
  os_name: IncusOS
  os_version: "202602230420"
  os_version_next: ""
  uptime: 63
```

### Wait for Updates

OC downloads IncusOS update packages from upstream. At least one update
must reach `ready` state before ISOs can be generated:

```bash
# Poll until at least one update shows "ready"
operations-center provisioning update list
```

Actual output (after ~8 minutes; UUIDs are stable across deployments):

| UUID | Origin | Channels | Version | Severity | Status |
|------|--------|----------|---------|----------|--------|
| 82aefab7-fec7-5122-89fd-8412d3d2174c | linuxcontainers.org | stable | 202602200553 | none | ready |
| 5d6b1018-e534-5e54-aeb5-c9e6027ab31d | linuxcontainers.org | stable | 202602210344 | none | ready |
| c912a390-c38b-5bd9-b46f-ccaeba6da68a | linuxcontainers.org | stable | 202602230420 | none | ready |

The table also includes `Upstream Channels` and `Published At` columns
(omitted for width). Not all updates may be ready simultaneously — at
least one `ready` is sufficient to proceed.

### Web UI Access

Open `https://192.168.102.120:8443/ui/` in your browser (with client.pfx
imported from Section 1). The web UI provides a dashboard view of the OC
server, update status, provisioning tokens, and system configuration.

---

## Section 3: Provision Nodes

### 3.1 Create Provisioning Token

```bash
operations-center provisioning token add --uses 5 --description "Production lab cluster"
operations-center provisioning token list
```

Actual output (UUID changes every run):

| UUID | Uses Remaining | Expire At | Channel | Description |
|------|----------------|-----------|---------|-------------|
| <token-UUID> | 5 | <30 days from now> | stable | Production lab cluster |

Save the `<token-UUID>` — you'll need it for the next steps.

### 3.2 Create Token Seed (No force_reboot)

**Critical**: the token seed must NOT include `force_reboot`. On Proxmox,
`incusos-proxmox` handles the install lifecycle externally (blockstat
detection + media removal). `force_reboot` triggers SysRq-B which causes
the crontab bug (~50% failure rate).

```yaml
# /tmp/oc-preseed.yaml
install:
  version: "1"
  force_install: true
```

**Important**: use the structured format with section keys (`install:`). A
flat format (`version: "1"` at root) maps fields to empty `{}` and they
don't get assigned to any section.

```bash
operations-center provisioning token seed add <token-UUID> proxmox-preseed \
    /tmp/oc-preseed.yaml --description "No force_reboot for Proxmox"
```

### 3.3 Generate OC-Provisioned ISO (Older Version)

**Critical discovery**: nodes deployed from an ISO matching the latest OC
update version are tracked as `needs_update: true` by OC because the OS
was never delivered through OC's update pipeline. The fix: generate the ISO
from an older channel so OC can push the real update after deployment.

```bash
# Create the old-stable channel (must exist before assigning updates to it)
operations-center provisioning channel add old-stable \
    --description "Older stable versions for initial provisioning"

# Assign the second-latest update to the old-stable channel
# (use the UUID for 202602210344 from `provisioning update list`)
operations-center provisioning update assign-channels <older-UUID> --channel old-stable

# Generate ISO from the older channel
operations-center provisioning token seed get-image <token-UUID> proxmox-preseed \
    /tmp/IncusOS-oc.iso --type iso --architecture x86_64 --channel old-stable
```

Actual output:

```
Successfully written 3433074688 bytes to "/tmp/IncusOS-oc.iso"
```

The ISO contains IncusOS 202602210344 (one version behind). OC will push
the latest (202602230420) after nodes register.

### 3.4 Node Configuration

```yaml
# incusos/examples/lab-oc-nodes.yaml
defaults:
  cores: 4
  memory: 20480
  disk: 100
  start_vmid: 400

proxmox:
  gateway: 192.168.100.1
  dns: 192.168.100.1

vms:
  - name: oc-node-01
    app: incus
    apply_defaults: false
    disk: 250
    ip: 192.168.102.140/22

  - name: oc-node-02
    app: incus
    apply_defaults: false
    ip: 192.168.102.141/22

  - name: oc-node-03
    app: incus
    apply_defaults: false
    ip: 192.168.102.142/22
```

**Key decisions**:

- **20 GiB RAM** per node: Aether needs 8 GiB, AWX needs 4-8 GiB
- **250 GiB disk** for node-01: hosts Aether's 200 GiB virtual image
- **100 GiB disk** for nodes 02-03: sufficient for AWX, monitoring, HAProxy
- **`apply_defaults: false`** for all nodes: OC's Terraform handles
  resource creation during cluster formation

### 3.5 Deploy Nodes (Hybrid Approach)

The hybrid approach uses `incusos-proxmox --iso` to combine OC
auto-registration (from the boot ISO token) with `incusos-proxmox` VM
creation, per-node SEED_DATA (hostname, static IP), install monitoring,
and media cleanup.

```bash
./incusos/incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes incusos/examples/lab-oc-nodes.yaml
```

Actual output (key lines):

```
[ok]  ISO uploaded: IncusOS-oc.iso
[ok]  VM 'oc-node-01' installed and running at 192.168.102.140
[ok]  Remote 'oc-node-01' added (192.168.102.140)
[ok]  VM 'oc-node-02' installed and running at 192.168.102.141
[ok]  Remote 'oc-node-02' added (192.168.102.141)
[ok]  VM 'oc-node-03' installed and running at 192.168.102.142
[ok]  Remote 'oc-node-03' added (192.168.102.142)
[ok]  All post-deployment checks passed
```

All 3 nodes: 876 MiB blockstat detection, clean install, no crontab bug.

### 3.6 Verify Auto-Registration

Nodes auto-register with OC within ~30 seconds of first boot. The update
from 202602210344 to 202602230420 happens automatically:

```bash
operations-center provisioning server list
```

Actual output (key columns; full table includes Type, Channel, Certificate
Fingerprint, Public Connection URL, Last Updated, Last Seen):

| Cluster | Name | Connection URL | Status | Update Status |
|---------|------|----------------|--------|---------------|
| | oc-node-01 | https://192.168.102.140:8443 | ready | up to date |
| | oc-node-02 | https://192.168.102.141:8443 | ready | up to date |
| | oc-node-03 | https://192.168.102.142:8443 | ready | up to date |
| | operations-center | https://[::1]:8443 | ready | update pending |

**Key**: all 3 nodes show **"up to date"** because OC delivered the
202602230420 update through its pipeline. This is what unlocks clustering.
Nodes may already be up to date by the time the last node finishes
deploying — the update gets pushed while `incusos-proxmox` deploys
subsequent nodes sequentially.

### 3.7 Verify Scrub Schedules

```bash
for node in oc-node-01 oc-node-02 oc-node-03; do
    incus query ${node}:/os/1.0/system/storage | python3 -c \
        "import sys,json; print('${node}:', json.load(sys.stdin)['config']['scrub_schedule'])"
done
```

Actual output:

```
oc-node-01: 0 4 * * 0
oc-node-02: 0 4 * * 0
oc-node-03: 0 4 * * 0
```

All healthy. No crontab bug (force_reboot was not used).

---

## Section 4: Form Cluster via Operations Center

### 4.1 The needs_update Blocker

OC requires all nodes to show `needs_update: false` before clustering.
Nodes deployed from an ISO matching the latest version are tracked as
`needs_update: true` because the OS was never delivered through OC's update
pipeline. The `needs_update` flag is server-side computed and cannot be
overridden via REST API PUT.

**Solution**: deploy from an older ISO version (Section 3.3). OC then pushes
the real update to nodes through its pipeline, clearing the flag.

### 4.2 Form Cluster

**Important**: if the client certificate was already injected via SEED_DATA,
use an empty application seed config to avoid "Certificate already in trust
store" Terraform errors:

```bash
echo '{}' > /tmp/oc-app-config.yaml

operations-center provisioning cluster add oc-cluster \
    https://192.168.102.140:8443 \
    --server-names oc-node-01,oc-node-02,oc-node-03 \
    --server-type incus \
    --application-seed-config /tmp/oc-app-config.yaml
```

OC orchestrates the full cluster formation:
1. Sets `core.https_address` to each node's specific IP
2. Enables clustering on oc-node-01
3. Joins oc-node-02 and oc-node-03
4. Creates storage pool (`local`), networks (`incusbr0`, `meshbr0`)
5. Runs Terraform/OpenTofu for post-cluster configuration

### 4.3 Fix Remotes After Clustering

Clustering regenerates TLS certificates. Re-add the remotes:

```bash
incus remote remove oc-node-01
incus remote remove oc-node-02
incus remote remove oc-node-03
incus remote add oc-node-01 https://192.168.102.140:8443 --accept-certificate
incus remote add oc-node-02 https://192.168.102.141:8443 --accept-certificate
incus remote add oc-node-03 https://192.168.102.142:8443 --accept-certificate
```

### 4.4 Verify Cluster

```bash
incus cluster list oc-node-01:
```

Actual output (key columns; full table includes FAILURE DOMAIN, DESCRIPTION):

| NAME | URL | ROLES | ARCHITECTURE | STATUS | MESSAGE |
|------|-----|-------|--------------|--------|---------|
| oc-node-01 | https://192.168.102.140:8443 | database-leader, database | x86_64 | ONLINE | Fully operational |
| oc-node-02 | https://192.168.102.141:8443 | database | x86_64 | ONLINE | Fully operational |
| oc-node-03 | https://192.168.102.142:8443 | database | x86_64 | ONLINE | Fully operational |

All 3 nodes ONLINE and Fully operational. The `ovn-chassis` role is added
later in Section 6.4.

### 4.5 Cluster Resources Created by OC

```bash
incus storage list oc-node-01:
incus network list oc-node-01:
```

Actual output (incusbr0 subnet varies per deployment):

| NAME | DRIVER | DESCRIPTION | USED BY | STATE |
|------|--------|-------------|---------|-------|
| local | zfs | Local storage pool (on system drive) | 8 | CREATED |

| NAME | TYPE | MANAGED | IPV4 | DESCRIPTION | USED BY |
|------|------|---------|------|-------------|---------|
| incusbr0 | bridge | YES | 10.x.x.1/24 | Local network bridge (NAT) | 1 |
| meshbr0 | bridge | YES | none | Internal mesh network bridge | 1 |

OC creates: `local` storage pool (ZFS), `incusbr0` bridge (NAT), and
`meshbr0` (OC-specific mesh network for inter-node communication). The
table also includes IPv6 and STATE columns.

---

## Section 5: Bridge Networking Baseline

Before setting up OVN, establish the baseline: bridge networks are
node-local. This demonstrates why OVN is needed.

### Same-Node Communication

Launch 2 containers on the same node. **Important**: use `--target` to force
placement — without it, the cluster scheduler may place containers on
different nodes automatically.

**Important**: launch containers **one at a time**, not chained with `&&`.
The first launch on a fresh cluster downloads the image (~1 GB), which
takes 2-3 minutes. Subsequent launches on the same node use the cached
image and are instant. Launches targeting a different node trigger another
image transfer to that node.

```bash
incus launch images:debian/12 oc-node-01:test-bridge-a --target oc-node-01
incus launch images:debian/12 oc-node-01:test-bridge-b --target oc-node-01
```

Wait for them to get IPs:

```bash
incus list oc-node-01: --columns ns4 --format csv | grep test-bridge
```

Ping between them:

```bash
IP_B=$(incus list oc-node-01:test-bridge-b --columns 4 --format csv | cut -d' ' -f1)
incus exec oc-node-01:test-bridge-a -- ping -c 3 "$IP_B"
```

Actual result: 0% packet loss, ~0.024ms latency. Same bridge, same node — works.

### Cross-Node Communication (Fails)

Launch a container on a different node:

```bash
incus launch images:debian/12 oc-node-01:test-bridge-c --target oc-node-02
```

Wait for IP:

```bash
incus list oc-node-01: --columns ns4 --format csv | grep test-bridge
```

Ping from node-01 to node-02:

```bash
IP_C=$(incus list oc-node-01:test-bridge-c --columns 4 --format csv | cut -d' ' -f1)
incus exec oc-node-01:test-bridge-a -- ping -c 3 -W 2 "$IP_C"
```

Actual result: **100% packet loss**. Bridge networks are node-local — there is
no L2 path between `incusbr0` on node-01 and `incusbr0` on node-02. Each
node's bridge has the same subnet (e.g., 10.251.22.1/24) but they are
separate L2 domains.

### Internet Access

NAT to the internet works from any node:

```bash
incus exec oc-node-01:test-bridge-a -- ping -c 3 1.1.1.1
```

Actual result: 0% packet loss, ~10ms latency. Each bridge provides NAT via
the host's management interface.

### Cleanup

```bash
incus delete oc-node-01:test-bridge-a --force
incus delete oc-node-01:test-bridge-b --force
incus delete oc-node-01:test-bridge-c --force
```


## Section 6: OVN Overlay Networking

OVN provides a cross-node L2 overlay using Geneve tunnels. After this
section, containers on any node can communicate transparently.

### 6.1 Deploy OVN Control Plane

Launch a Debian container on node-01 to host the OVN central services:

```bash
incus launch images:debian/12 oc-node-01:ovn-central --target oc-node-01
```

Install OVN:

```bash
incus exec oc-node-01:ovn-central -- bash -c \
    "apt-get update -qq && apt-get install -y -qq ovn-central"
```

Configure OVN to listen on all interfaces:

```bash
incus exec oc-node-01:ovn-central -- ovn-nbctl set-connection ptcp:6641:0.0.0.0
incus exec oc-node-01:ovn-central -- ovn-sbctl set-connection ptcp:6642:0.0.0.0
```

Add proxy devices to expose NB (6641) and SB (6642) on the host's LAN IP:

```bash
incus config device add oc-node-01:ovn-central \
    nb-proxy proxy listen=tcp:192.168.102.140:6641 connect=tcp:127.0.0.1:6641
incus config device add oc-node-01:ovn-central \
    sb-proxy proxy listen=tcp:192.168.102.140:6642 connect=tcp:127.0.0.1:6642
```

### 6.2 Enable OVN on All IncusOS Nodes

Enable OVN services via the IncusOS REST API (`/os/1.0/services/ovn`).
The `database` field is the **southbound** DB (port 6642), not northbound.

```bash
for node_ip in 192.168.102.140 192.168.102.141 192.168.102.142; do
    remote="oc-node-$(echo $node_ip | cut -d. -f4 | sed 's/140/01/;s/141/02/;s/142/03/')"
    incus query ${remote}:/os/1.0/services/ovn --request PUT --data "{
        \"config\": {
            \"database\": \"tcp:192.168.102.140:6642\",
            \"enabled\": true,
            \"tunnel_address\": \"${node_ip}\",
            \"tunnel_protocol\": \"geneve\"
        },
        \"state\": {}
    }"
done
```

Each call should return `{}` on success.

### 6.3 Configure Incus OVN Connection

Point Incus to the **northbound** DB (port 6641):

```bash
incus config set oc-node-01: network.ovn.northbound_connection tcp:192.168.102.140:6641
```

### 6.4 Assign OVN Chassis Role

Every node that will host OVN workloads needs the `ovn-chassis` role:

```bash
for node in oc-node-01 oc-node-02 oc-node-03; do
    incus cluster role add oc-node-01:${node} ovn-chassis
done
```

Verify:

```bash
incus cluster list oc-node-01:
```

The ROLES column should now include `ovn-chassis` for each member.

### 6.5 Create UPLINK Physical Network

The UPLINK network provides the bridge between OVN virtual networks and
the physical LAN. It uses the two-step cluster pattern: per-member
`--target` first, then cluster-wide create.

**Important**: IncusOS names its management NIC `mgmt`, NOT `ens18`. Using
`parent=ens18` will fail with "Parent interface 'ens18' not found".

```bash
# Per-target (parent is member-specific)
for node in oc-node-01 oc-node-02 oc-node-03; do
    incus network create oc-node-01:UPLINK --type=physical --target=${node} parent=mgmt
done

# Cluster-wide config
incus network create oc-node-01:UPLINK --type=physical \
    ipv4.ovn.ranges=192.168.103.200-192.168.103.210 \
    ipv4.gateway=192.168.100.1/22 \
    dns.nameservers=192.168.100.1
```

### 6.6 Create OVN Network (net-prod)

```bash
incus network create oc-node-01:net-prod --type=ovn \
    network=UPLINK ipv4.address=10.10.10.1/24 ipv4.nat=true
```

Actual output:

```
Network net-prod created
```

net-prod is assigned external IP `192.168.103.200` from the UPLINK range.

### 6.7 Verify Cross-Node OVN Connectivity

```bash
incus launch images:debian/12 oc-node-01:test-1 --target oc-node-01 -n net-prod
incus launch images:debian/12 oc-node-01:test-2 --target oc-node-02 -n net-prod
incus exec oc-node-01:test-1 -- ping -c 3 10.10.10.3
```

Actual output:

```
64 bytes from 10.10.10.3: icmp_seq=1 ttl=64 time=0.669 ms
64 bytes from 10.10.10.3: icmp_seq=2 ttl=64 time=0.136 ms
64 bytes from 10.10.10.3: icmp_seq=3 ttl=64 time=0.194 ms
```

Sub-millisecond cross-node latency via Geneve tunnels. Clean up test
containers after verification:

```bash
incus delete oc-node-01:test-1 oc-node-01:test-2 --force
```

### 6.8 Final Network State

```bash
incus network list oc-node-01:
```

Actual output:

| NAME | TYPE | MANAGED | IPV4 | DESCRIPTION | USED BY |
|------|------|---------|------|-------------|---------|
| UPLINK | physical | YES | | | 1 |
| incusbr0 | bridge | YES | 10.x.x.1/24 | Local network bridge (NAT) | 2 |
| meshbr0 | bridge | YES | none | Internal mesh network bridge | 1 |
| net-prod | ovn | YES | 10.10.10.1/24 | | 0 |

The incusbr0 subnet is randomly assigned per deployment. The USED BY count
for net-prod is 0 at this point (test containers deleted); it increases as
workloads are added in subsequent sections.

---

## Section 7: Deploy Aether

Aether is the management platform for the Incus cluster. It provides
instance deployment via blueprints, HAProxy load balancing, AWX automation
integration, firewall rule management, and RBAC — all through a web UI
and partial REST API.

### 7.1 Import Golden Image

The golden image is a Ubuntu Noble (24.04) image with the Aether
application pre-installed. Import it to the cluster:

```bash
incus image import sources/aether-golden-image-v6.tar.gz \
    --alias aether-golden-image-v6 oc-node-01:
```

This takes ~2 minutes. The image is ~6.6 GiB compressed with a 200 GiB
virtual disk.

### 7.2 Create and Configure VM

Use `incus init` (not `launch`) — the VM needs NIC configuration before
first boot:

```bash
incus init oc-node-01:aether-golden-image-v6 oc-node-01:aether --vm \
    --target oc-node-01 --config limits.memory=8GiB -d root,size=200GiB
```

**Important**: node-01 must have a 250 GiB disk (configured in Section 3).
The 200 GiB virtual disk is thin-provisioned (~11 GiB actual data), but the
storage pool needs 200 GiB of allocatable space.

Switch the NIC from bridge to macvlan for direct VLAN access:

```bash
incus config device remove oc-node-01:aether eth0 2>/dev/null || true
incus config device add oc-node-01:aether eth0 nic nictype=macvlan parent=mgmt
```

### 7.3 Start and Run Post-Deploy

```bash
incus start oc-node-01:aether
```

Wait for the VM agent (~30-60s), then run the post-deploy script. This
configures static networking, regenerates SSH keys, initializes the
database, and starts the Aether service:

```bash
# Wait for agent
sleep 30

incus exec oc-node-01:aether -- /home/ffsdn/post_deploy.sh \
    192.168.102.160/22 192.168.100.1 192.168.100.1
```

Expected output (7 configuration steps):

```
Step 1: Configuring network...
Step 2: Applying network configuration...
Step 3: Regenerating SSH host keys...
Step 4: Waiting for FFSDN to create PostgreSQL user...
Step 5: Transferring database ownership...
Step 6: Transferring table and sequence ownership...
Step 7: Restarting FFSDN service...
Post-deploy completed successfully.
```

The script deletes itself after successful execution.

### 7.4 Set Admin Password

Access the Aether web UI at `https://192.168.102.160:8443`. The initial
setup prompts for an admin password. Set it to the value in your `env`
file (`AETHER_ADMIN_PASSWORD`).

### 7.5 Connect Incus Cluster

Generate a trust token on the Incus cluster:

```bash
incus config trust add oc-node-01:AETHER
```

In the Aether web UI:

1. Navigate to **Manage INCUS Clusters** (`/incus-infra`)
2. Click **Add/Edit/Delete INCUS clusters from/to AETHER**
3. Fill in the form:
   - **Cluster Name**: `oc-lab-cluster`
   - **URL**: `https://192.168.102.141:8443`
   - **Trust Token**: paste the token from above
4. Click **Add Cluster**

**Note**: use any cluster member's IP as the URL. Aether connects to the
cluster through this endpoint and discovers all members automatically.

After adding, select the cluster from the dropdown on the INCUS
Infrastructure Management page. You should see all 3 nodes, their
status, memory usage, and OVN roles.

### 7.6 Connect Operations Center

In the Aether web UI:

1. Navigate to **Operations Center** (`/operationcenter`)
2. Click **+ Add Operations Center**
3. Fill in the form:
   - **Name**: `oc-lab`
   - **URL**: `https://192.168.102.120:8443`
   - **Certificate (PEM)**: contents of `~/.config/incus/client.crt`
   - **Private Key (PEM)**: contents of `~/.config/incus/client.key`
4. Click **Test Connection** — expect: `Connected! API: 1.0 (devel)`
5. Click **Save**

### 7.7 Verify

```bash
# Get a JWT token and query clusters
curl -sk https://192.168.102.160:8443/api/clusters \
    -H "Authorization: Bearer $(curl -sk -X POST \
    https://192.168.102.160:8443/api/auth/token \
    -H 'Content-Type: application/json' \
    -d '{"username":"admin","password":"'"$(grep AETHER_ADMIN_PASSWORD env \
        | cut -d= -f2 | tr -d \"\')"'"}' \
    | python3 -c "import sys,json; print(json.load(sys.stdin)['token'])")" \
    | python3 -m json.tool
```

Expected: JSON array containing `oc-lab-cluster` with cluster members,
ID, and connection status.


## Section 8: HAProxy HA Load Balancing

Deploy an HA load balancer pair with test backends using the
`deploy-haproxy` script. This automates Aether's HAProxy management
features: image build, infrastructure deployment, and service creation.

### 8.1 Prerequisites Check

```bash
cd incusos
./deploy-haproxy --doctor
```

Expected output:

```
[step]    Checking prerequisites
[ok]      incus CLI: 6.21
[ok]      curl: 8.x.x
[ok]      python3: 3.x.x
[ok]      Aether password: configured
[ok]      Incus remote 'oc-node-01': reachable
[ok]      Aether: reachable at https://192.168.102.160:8443 (HTTP 200)
[info]    No HAProxy infrastructure on cluster 52 (--deploy to create)
[ok]      All prerequisites satisfied
```

### 8.2 Preview

Preview the full deployment without executing:

```bash
./deploy-haproxy --deploy --dry-run
```

This shows all actions that would be taken: backend creation, image build,
infrastructure deployment, and service configuration.

### 8.3 Deploy

```bash
./deploy-haproxy --deploy
```

The script runs 5 phases:

1. **Deploy test backends**: 3 nginx containers (nginx-lb-01/02/03) on
   net-prod at 10.10.10.60-62, each serving a unique HTML page
2. **Build HAProxy image**: builds a HAProxy container image via Aether's
   session API (5-10 minutes on first run, cached afterwards)
3. **Deploy infrastructure**: creates the HA pair (ffsdn-haproxy-52-01/02)
   on net-prod at 10.10.10.50/51, with an OVN load balancer VIP at
   192.168.103.200
4. **Create service**: configures the `web-test` HTTP service with
   health checks and round-robin balancing across the 3 backends
5. **Verify**: tests VIP connectivity and traffic distribution

Full deployment takes 10-15 minutes (dominated by image build on first
run).

### 8.4 Verify Load Balancing

From your dev machine (must have routing to 192.168.103.0/24):

```bash
for i in $(seq 1 6); do curl -s http://192.168.103.200; echo; done
```

Expected output shows responses from the 3 backends:

```html
<html><body>
<h1>Backend: nginx-lb-01</h1>
<p>IP: 10.10.10.60</p>
</body></html>
```

Traffic distribution across multiple backends confirms HA load balancing
is working. OVN uses connection-based hashing — sequential requests from
the same source may hit the same backend.

### 8.5 Health and Status

```bash
./deploy-haproxy --status
```

Shows HAProxy container health, backend status, VIP connectivity, and
Aether service health stats including per-backend UP/DOWN status and
current session count.

### 8.6 ACL Notes

Aether creates per-container ACLs with default-deny rules. The deploy
script configures services to work within this model. If external access
to the VIP doesn't work, check the HAProxy containers' ACLs:

```bash
incus network acl list oc-node-01:
```

Look for ACLs named `52-ffsdn-haproxy-52-01-aether-acl` and similar. The
script's service creation should have added the necessary ingress rules
for backend traffic.


## Section 9: AWX Lifecycle Automation

Deploy AWX (the open-source Ansible Tower) on a Debian 12 VM with K3s,
configure lifecycle playbooks, and register with Aether for automated
post-deploy and decommission hooks.

### 9.1 Prerequisites Check

```bash
./deploy-awx --doctor
```

### 9.2 Deploy

```bash
./deploy-awx --deploy
```

The script runs 6 phases:

1. **Create VM**: Debian 12 with 4 vCPU, 8 GiB RAM, 40 GiB disk on
   oc-node-02 with macvlan NIC
2. **Configure network**: static IP 192.168.102.161/22
3. **Install K3s**: lightweight Kubernetes (`--write-kubeconfig-mode 644`)
4. **Deploy AWX Operator**: from kustomize manifests in `awx-manifests/`
5. **Deploy AWX instance**: web UI on NodePort 30080, task engine,
   PostgreSQL
6. **Verify**: API ping, admin password retrieval

Full deployment takes 15-20 minutes. AWX is accessible via HTTP (not
HTTPS) at port 30080 — Traefik's ingress returns 404 for IP-based access.

### 9.3 Configure

Set up the AWX project, inventory, credentials, and job templates:

```bash
./deploy-awx --configure
```

This creates:

- **Project**: `incus-contrib` (manual/local_path — playbooks are pushed
  directly to the AWX task pod since the private Git repo is not reachable
  from the Execution Environment container)
- **Inventory**: `incus-instances`
- **Credential**: `incus-instances` (machine type — required by AWX
  templates, though the playbooks use Incus API not SSH)
- **Templates**: `post-deploy` and `decommission`

The Incus client cert/key are pushed to the AWX task pod at
`/var/lib/awx/projects/incus-contrib/`. During job execution, the EE
container mounts this as `/runner/project/` — the playbooks reference
certs at that path.

### 9.4 Register with Aether

```bash
./deploy-awx --join-aether
```

This creates a Personal Access Token (PAT) in AWX and prints the manual
steps to complete Aether integration. In the Aether web UI:

1. Navigate to **Settings → Ansible Automation** (`/awx-endpoints`)
2. **Add Endpoint**:
   - **Name**: `lab-awx`
   - **URL**: `http://192.168.102.161:30080`
   - **Token**: the PAT printed by the script
   - **Verify SSL**: unchecked
3. **Configure cluster AWX binding** (from the cluster settings):
   - **Post-deploy template ID**: printed by the script
   - **Decommission template ID**: printed by the script
   - **Job timeout**: 600 seconds

### 9.5 How the Lifecycle Hooks Work

When Aether creates an instance, it triggers the AWX **post-deploy**
template with `ffsdn_*` variables (instance name, IP, cluster, deployer,
image info). The playbook:

1. Waits for the instance to reach Running state
2. Pushes a setup script via Incus file API
3. Executes it via Incus exec API (sets hostname, installs base packages,
   configures unattended upgrades, writes deployment metadata)
4. Verifies success

When Aether deletes an instance, it triggers the **decommission** template.
The playbook gracefully stops application services and records the event
in a ledger.

**Important**: post-deploy failures cause Aether to auto-rollback (delete
the instance). Decommission failures do NOT block deletion.

### 9.6 Verify

```bash
# AWX API health
curl -s http://192.168.102.161:30080/api/v2/ping/ | python3 -m json.tool

# Full status
./deploy-awx --status
```

The status output shows VM health, K3s state, pod status, admin password,
AWX version, and Aether connectivity.


## Section 10: Observability Stack

Deploy Prometheus, Grafana, Loki, Promtail, and node-exporters for full
cluster observability. The stack scrapes Incus metrics (with TLS client
certs), HAProxy stats, and host-level metrics.

### 10.1 Deploy

```bash
./deploy-observability --deploy
```

The script runs 9 phases:

1. **Monitoring container**: Debian 12, 2 GiB RAM, 20 GiB disk on
   oc-node-02 at 10.10.10.70
2. **Prometheus**: scrapes Incus nodes (`:8443/1.0/metrics` with client
   certs), HAProxy stats (`:8404`), and node-exporters (`:9100`)
3. **Loki + Promtail**: log aggregation from systemd journal and
   `/var/log`
4. **Grafana**: dashboards with Prometheus and Loki datasources
5. **Node-exporters**: 3 Alpine containers (node-exp-01/02/03) with host
   filesystem bind-mounts for full host metrics
6. **ACL**: `monitoring-allow` ACL on all observability containers
7. **HAProxy exporter**: prometheus exporter route added to HAProxy
   containers, plus monitoring ACL rules for scrape access on port 8404
8. **Network forward**: 192.168.103.201 → Grafana (`:3000`) and
   Prometheus (`:9090`)
9. **Dashboards**: pre-built JSON dashboards uploaded to Grafana

Full deployment takes ~5 minutes.

### 10.2 Access Dashboards

- **Grafana**: `http://192.168.103.201:3000` (admin / admin)
- **Prometheus**: `http://192.168.103.201:9090`

Pre-loaded dashboards:

| Dashboard | Description |
|-----------|-------------|
| Incus Cluster Overview | Instances, memory, CPU across all 3 nodes |
| HAProxy Traffic | Request rate, backend health, error rates |
| Host Resources | CPU, memory, disk, network per node |

### 10.3 Verify

```bash
./deploy-observability --status
```

Shows container health, service status (prometheus, grafana-server, loki,
promtail), Prometheus target scrape results, Grafana API health, Loki
ingestion status, and network forward connectivity.

For detailed diagnostics:

```bash
./deploy-observability --doctor
```

This checks container networking (gateway, node-exporter, HAProxy
reachability), Prometheus scrape errors, service health with log output
on failure, and disk space.


## Section 11: Network Isolation & Security

### 11.1 Create Isolated Network

```bash
incus network create oc-node-01:net-isolated --type=ovn network=UPLINK \
    ipv4.address=10.10.20.1/24 \
    ipv4.nat=true \
    ipv6.address=none
```

### 11.2 Launch Isolated Containers

```bash
incus launch images:debian/12 oc-node-01:iso-app-01 --network net-isolated --target oc-node-01
incus launch images:debian/12 oc-node-01:iso-app-02 --network net-isolated --target oc-node-02
```

### 11.3 Verify Network Isolation

Containers on net-isolated can reach each other:

```bash
IP_ISO2=$(incus list oc-node-01:iso-app-02 --columns 4 --format csv | cut -d' ' -f1)
incus exec oc-node-01:iso-app-01 -- ping -c 3 "$IP_ISO2"
```

Actual result: 0% packet loss, ~0.15-0.5ms latency. Containers on the
same OVN network can reach each other across nodes.

But net-prod **cannot** reach net-isolated:

```bash
incus exec oc-node-01:nginx-lb-01 -- ping -c 3 -W 2 "$IP_ISO2"
```

Actual result: **100% packet loss**. Different OVN networks are fully
isolated — separate L2 domains, no routing between them.

### 11.4 Create Network ACL

Create an ACL that blocks ICMP from a specific source:

```bash
incus network acl create oc-node-01:block-ping
incus network acl rule add oc-node-01:block-ping ingress \
    action=drop protocol=icmp4 \
    source=10.10.20.0/24 \
    description="Block ICMP within net-isolated"
```

### 11.5 Apply and Test ACL

Apply the ACL to net-isolated:

```bash
incus network set oc-node-01:net-isolated security.acls=block-ping
```

Verify ICMP is blocked:

```bash
incus exec oc-node-01:iso-app-01 -- ping -c 3 -W 2 "$IP_ISO2"
```

Remove the ACL:

```bash
incus network unset oc-node-01:net-isolated security.acls
```

Verify ICMP works again:

```bash
incus exec oc-node-01:iso-app-01 -- ping -c 3 "$IP_ISO2"
```

### 11.6 Network Peering

Connect net-prod and net-isolated so containers on both networks can
communicate. Peering is bilateral — create a peer on both sides:

```bash
# From net-prod's perspective
incus network peer create oc-node-01:net-prod peer-to-isolated net-isolated \
    --description "Peer to isolated network"

# From net-isolated's perspective
incus network peer create oc-node-01:net-isolated peer-to-prod net-prod \
    --description "Peer to production network"
```

### 11.7 Verify Peering

Cross-network ping (prod → isolated):

```bash
incus exec oc-node-01:nginx-lb-01 -- ping -c 3 "$IP_ISO2"
```

Actual result: 0% packet loss with TTL=62 (64 - 2 router hops),
confirming traffic traverses the OVN routers on both sides of the peering.

Cross-network ping (isolated → prod):

```bash
NLB1_IP=$(incus list oc-node-01:nginx-lb-01 --columns 4 --format csv | cut -d' ' -f1)
incus exec oc-node-01:iso-app-01 -- ping -c 3 "$NLB1_IP"
```

### 11.8 Remove Peering

```bash
incus network peer delete oc-node-01:net-prod peer-to-isolated
incus network peer delete oc-node-01:net-isolated peer-to-prod
```

Verify isolation is restored:

```bash
incus exec oc-node-01:nginx-lb-01 -- ping -c 3 -W 2 "$IP_ISO2"
```

Expected: 100% packet loss. Networks are isolated again.

Clean up isolated containers:

```bash
incus delete oc-node-01:iso-app-01 --force
incus delete oc-node-01:iso-app-02 --force
```


## Section 12: Live Migration

### 12.1 Deploy Test VM

Deploy a temporary VM on net-prod for migration testing. Live migration
requires VMs (containers use a different migration path).

```bash
incus launch images:debian/12 oc-node-01:migration-test --vm \
    --network net-prod --target oc-node-01
```

Wait for boot (~30-60s), then configure for stateful migration:

```bash
incus stop oc-node-01:migration-test
incus config set oc-node-01:migration-test limits.cpu=0-1
incus config set oc-node-01:migration-test migration.stateful=true
incus config device override oc-node-01:migration-test root size.state=2GiB
incus start oc-node-01:migration-test
```

**Critical**: `limits.cpu` must be a **range** (e.g., `0-1`), not an
integer. Without the range, QEMU sets `maxcpus` based on the host's CPU
count, which varies across nodes and breaks migration with
`Missing section footer for ICH9LPC`.

**Important**: use `device override` (not `device set`) because the `root`
device comes from the default profile.

### 12.2 Create Heartbeat Service

Create a simple counter to verify state continuity across migration:

```bash
incus exec oc-node-01:migration-test -- bash -c '
    mkdir -p /tmp/heartbeat
    nohup bash -c "i=0; while true; do echo \$i > /tmp/heartbeat/counter; i=\$((i+1)); sleep 1; done" \
        > /dev/null 2>&1 &
    echo "Heartbeat started"
'
```

Read the counter:

```bash
incus exec oc-node-01:migration-test -- cat /tmp/heartbeat/counter
```

Note the value. After migration, the counter should continue from where
it left off (live migration preserves running state).

### 12.3 Live Migration Round-Trip

Check current location:

```bash
incus list oc-node-01:migration-test --columns nL --format csv
```

**Migrate node-01 → node-02:**

```bash
time incus move oc-node-01:migration-test --target oc-node-02
```

Actual result: ~7s (~140 MB/s). Wait for the VM agent to reconnect:

```bash
sleep 4
incus exec oc-node-01:migration-test -- cat /tmp/heartbeat/counter
```

The heartbeat counter continues incrementing — the process was never
interrupted during migration.

**Migrate node-02 → node-03:**

```bash
time incus move oc-node-01:migration-test --target oc-node-03
sleep 4
incus exec oc-node-01:migration-test -- cat /tmp/heartbeat/counter
```

**Migrate node-03 → node-01 (back to origin):**

```bash
time incus move oc-node-01:migration-test --target oc-node-01
sleep 4
incus exec oc-node-01:migration-test -- cat /tmp/heartbeat/counter
```

Verify the VM is back on node-01:

```bash
incus list oc-node-01:migration-test --columns nL --format csv
```

### 12.4 Stateful Stop/Restore

Stateful stop saves VM memory to disk. On start, the VM resumes exactly
where it was:

```bash
# Note the heartbeat counter
incus exec oc-node-01:migration-test -- cat /tmp/heartbeat/counter
```

Stateful stop:

```bash
incus stop oc-node-01:migration-test --stateful
```

Start (resumes from saved state):

```bash
incus start oc-node-01:migration-test
sleep 4
incus exec oc-node-01:migration-test -- cat /tmp/heartbeat/counter
```

The counter resumes from the saved value — the entire VM state is
preserved across stateful stop/start cycles.

**If the restore fails** (e.g., from a `limits.cpu` mismatch), discard the
saved state:

```bash
incus start oc-node-01:migration-test --stateless
```

### 12.5 Clean Up Test VM

```bash
incus delete oc-node-01:migration-test --force
```


## Section 13: Cluster Lifecycle

### 13.1 Evacuation & Restore

Evacuate node-02. All workloads are moved to other nodes:

```bash
incus cluster evacuate oc-node-01:oc-node-02 --force
```

Check workload distribution — nothing on node-02:

```bash
incus list oc-node-01: --columns nstL --format table
```

Actual behavior: containers are stopped, moved to other nodes, then
restarted. The `--force` flag skips confirmation prompts.

Verify node-02 shows EVACUATED:

```bash
incus cluster list oc-node-01:
```

Restore node-02 (workloads return):

```bash
incus cluster restore oc-node-01:oc-node-02 --force
```

Verify all workloads are back:

```bash
incus list oc-node-01: --columns nstL --format table
incus cluster list oc-node-01:
```

All nodes should show ONLINE.

### 13.2 Node Failure Simulation

A Proxmox hard-stop on a VM simulates a crash. The Incus cluster heartbeat
detects the failure in ~40 seconds. After the node is restarted:

1. The node auto-rejoins the cluster (~60s)
2. Containers auto-start
3. VMs that were running resume

**Procedure** (document only — do not execute while OVN is running unless
you can tolerate temporary network disruption):

```bash
# Simulate crash: hard-stop via Proxmox API
# incusos/helpers/proxmox-api POST /nodes/pve/qemu/401/status/stop

# Wait for heartbeat detection (~40s)
# incus cluster list oc-node-01:
# → oc-node-02 shows OFFLINE

# Restart via Proxmox
# incusos/helpers/proxmox-api POST /nodes/pve/qemu/401/status/start

# Wait for auto-rejoin (~60s)
# incus cluster list oc-node-01:
# → oc-node-02 shows ONLINE
```

### 13.3 Node Replacement

Full procedure: evacuate a node, remove it from the cluster, destroy the
VM, deploy a fresh node, and join it back. This tests the complete lifecycle.

**Step 1: Evacuate node-03:**

```bash
incus cluster evacuate oc-node-01:oc-node-03 --force --action stop
```

**Step 2: Remove from cluster:**

```bash
printf "yes\n" | incus cluster remove oc-node-01:oc-node-03 --force
```

**Note**: `incus cluster remove` prompts "Are you really sure?" even with
`--force`. The `printf` pipes `yes` for automation.

**Step 3: Clean up the remote:**

```bash
incus remote remove oc-node-03
```

**Step 4: Destroy and redeploy the VM.** Use `incusos-proxmox` to destroy
just node-03 (VMID 402) and redeploy it. Create a single-VM config:

```yaml
# /tmp/lab-replace-node03.yaml
defaults:
  cores: 4
  memory: 20480
  disk: 100
  start_vmid: 402

vms:
  - name: oc-node-03
    app: incus
    apply_defaults: false
    ip: 192.168.102.142/22
```

```bash
./incusos-proxmox --cleanup --yes /tmp/lab-replace-node03.yaml
./incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes /tmp/lab-replace-node03.yaml
```

**Step 5: Join the fresh node to the cluster:**

```bash
# Set specific IP
incus config set oc-node-03: core.https_address 192.168.102.142:8443

# Generate join token
incus cluster add oc-node-01:oc-node-03

# Join
printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join oc-node-01: oc-node-03:

# Fix remote
incus remote remove oc-node-03
incus remote add oc-node-03 https://192.168.102.142:8443 --accept-certificate
```

**Step 6: Re-enable OVN on the replacement node:**

```bash
incus query oc-node-03:/os/1.0/services/ovn --request PUT --data '{
  "config": {
    "database": "tcp:192.168.102.140:6642",
    "enabled": true,
    "tunnel_address": "192.168.102.142",
    "tunnel_protocol": "geneve"
  },
  "state": {}
}'

incus cluster role add oc-node-01:oc-node-03 ovn-chassis
```

**Step 7: Verify:**

```bash
incus cluster list oc-node-01:
```

All 3 nodes should be ONLINE with `ovn-chassis` role.

### 13.4 Cluster Rebalancing

Enable automatic workload rebalancing. When a new node joins (or workloads
are unevenly distributed), Incus redistributes VMs:

```bash
incus config set oc-node-01: cluster.rebalance.interval=1
incus config set oc-node-01: cluster.rebalance.threshold=10
incus config set oc-node-01: cluster.rebalance.batch=2
incus config set oc-node-01: cluster.rebalance.cooldown=5m
```

**Important**: only VMs with `migration.stateful=true` are rebalanced.
Containers are NOT auto-rebalanced.

Monitor rebalancing:

```bash
incus list oc-node-01: --columns nstL --format table
```

Disable rebalancing when done testing:

```bash
incus config unset oc-node-01: cluster.rebalance.interval
incus config unset oc-node-01: cluster.rebalance.threshold
```


## Section 14: OC Dashboard

**Important**: Switch to the OC remote first. The OC CLI does not support
`remote:` suffix syntax:

```bash
operations-center remote switch oc-lab
```

### OC Server Information

```bash
operations-center admin os show
```

Actual output (version and uptime will vary):

```
environment:
  hostname: oc-server
  os_name: IncusOS
  os_version: "202602240349"
  os_version_next: ""
  uptime: 3600
```

### Provisioning Status

OC manages all 3 cluster nodes. Verify they're registered and up to date:

```bash
operations-center provisioning server list
```

All nodes should show `ready` status and `up to date` update status.

### Web UI

The OC web UI at `https://192.168.102.120:8443/ui/` provides:

- **Dashboard**: server overview with resource utilization
- **Updates**: available IncusOS updates and delivery status
- **Provisioning**: token management, server list, cluster formation
- **System**: OC configuration and certificates

Because the nodes were deployed from an OC-provisioned ISO (Section 3),
OC has full visibility and management of the cluster — including update
delivery, server inventory, and cluster formation.


## Section 15: Cleanup

### Tear Down Services (Reverse Order)

Tear down deployed services in reverse order. Each script includes an
interactive confirmation prompt:

```bash
cd incusos

# 1. Observability stack
./deploy-observability --cleanup

# 2. AWX VM
./deploy-awx --cleanup

# 3. HAProxy infrastructure + backends
./deploy-haproxy --cleanup
```

### Delete Aether VM

```bash
incus stop oc-node-01:aether --force
incus delete oc-node-01:aether --force
incus image delete oc-node-01:aether-golden-image-v6 2>/dev/null || true
```

### Remove Isolated Network (if created)

```bash
incus network delete oc-node-01:net-isolated 2>/dev/null || true
incus network acl delete oc-node-01:block-ping 2>/dev/null || true
```

### Remove OVN Networks

```bash
incus network delete oc-node-01:net-prod
incus network delete oc-node-01:UPLINK
```

### Remove OVN Control Plane

```bash
incus delete oc-node-01:ovn-central --force
```

### Disable OVN Services

```bash
for node in oc-node-01 oc-node-02 oc-node-03; do
    incus query "$node":/os/1.0/services/ovn --request PUT --data '{
      "config": {
        "enabled": false
      },
      "state": {}
    }'
done
```

### Infrastructure Options

**Keep infrastructure** (stop VMs, keep on disk for later):

```bash
./incusos-proxmox --lab-down examples/lab-oc-nodes.yaml
./incusos-proxmox --lab-down examples/lab-oc-deploy.yaml
```

Restart later with:

```bash
./incusos-proxmox --lab-up examples/lab-oc-deploy.yaml
./incusos-proxmox --lab-up examples/lab-oc-nodes.yaml
```

**Full teardown** (destroy all VMs, remove ISOs, remotes, cache):

```bash
./incusos-proxmox --cleanup-all --deep --yes
```


## Section 16: Verification Checklist

| # | Check | Command | Expected |
|---|-------|---------|----------|
| 1 | All VMs running | `incusos-proxmox --status examples/lab-oc-nodes.yaml` | 4 VMs running, port 8443 open |
| 2 | Scrub schedule | `incus query oc-node-01:/os/1.0/system/storage` | `scrub_schedule: "0 4 * * 0"` |
| 3 | OC accessible | `operations-center admin os show` | Shows version, uptime |
| 4 | Cluster formed | `incus cluster list oc-node-01:` | 3 nodes ONLINE |
| 5 | OVN connectivity | Ping cross-node on net-prod | 0% loss |
| 6 | Aether web UI | `https://192.168.102.160:8443` | Login page loads |
| 7 | Aether API | `GET /api/clusters` with JWT | Returns cluster data |
| 8 | HAProxy VIP | `curl http://192.168.103.200` | Backend response |
| 9 | HAProxy HA | `./deploy-haproxy --status` | Both containers running |
| 10 | AWX API | `curl http://192.168.102.161:30080/api/v2/ping/` | JSON with version |
| 11 | AWX templates | `./deploy-awx --status` | Pods running, templates configured |
| 12 | Grafana | `http://192.168.103.201:3000` | Dashboard loads |
| 13 | Prometheus targets | `./deploy-observability --status` | All targets UP |
| 14 | Network isolation | Ping net-prod → net-isolated | 100% loss (expected) |
| 15 | Network peering | Peer + ping cross-network | 0% loss, TTL=62 |
| 16 | VM live migration | `incus move` VM between nodes | State preserved |
| 17 | Cluster evacuation | `incus cluster evacuate` + `restore` | Workloads moved and returned |
| 18 | Stateful stop/start | `incus stop --stateful` + `start` | VM state preserved |
| 19 | DNS resolution | `dig hostname.incus` from container | Resolves to 10.10.10.x |


## Section 17: Quick Reference

### Service Endpoints

| Service | URL | Auth |
|---------|-----|------|
| Aether | `https://192.168.102.160:8443` | admin / (env file) |
| AWX | `http://192.168.102.161:30080` | admin / (from K8s secret) |
| Grafana | `http://192.168.103.201:3000` | admin / admin |
| Prometheus | `http://192.168.103.201:9090` | none |
| HAProxy VIP | `http://192.168.103.200` | none |
| OC Web UI | `https://192.168.102.120:8443/ui/` | client certificate |

### Deploy Script Commands

| Script | Action | Description |
|--------|--------|-------------|
| `deploy-haproxy --doctor` | Prerequisites | Check tools, Aether, cluster |
| `deploy-haproxy --deploy` | Full deploy | Backends + image + infra + service |
| `deploy-haproxy --status` | Health check | Container + VIP + backend status |
| `deploy-haproxy --cleanup` | Tear down | Remove all HAProxy resources |
| `deploy-awx --deploy` | Full deploy | VM + K3s + AWX operator + instance |
| `deploy-awx --configure` | Configure | Project + inventory + templates |
| `deploy-awx --join-aether` | Register | Create PAT, print Aether steps |
| `deploy-awx --status` | Health check | VM + K3s + pods + API |
| `deploy-awx --heal` | Fix issues | Restart pods, sync project |
| `deploy-awx --cleanup` | Tear down | Destroy AWX VM |
| `deploy-observability --deploy` | Full deploy | All 9 phases |
| `deploy-observability --status` | Health check | Services + targets + forward |
| `deploy-observability --doctor` | Diagnose | Network + scrape + disk |
| `deploy-observability --cleanup` | Tear down | Remove all observability |

### Cluster Command Syntax

| Command | Arguments | Notes |
|---------|-----------|-------|
| `incus cluster enable` | `remote: member-name` | TWO args (space between) |
| `incus cluster add` | `remote:member-name` | ONE arg (no space) |
| `incus cluster join` | `init-remote: joining-remote:` | TWO args (space between) |
| `incus cluster remove` | `remote:member-name --force` | ONE arg; prompts even with `--force` |
| `incus cluster evacuate` | `remote:member-name` | ONE arg (no space) |
| `incus cluster restore` | `remote:member-name` | ONE arg (no space) |
| `incus config set` | `remote: key value` | Remote with trailing colon + space |
| `incus storage show` | `remote:pool` | ONE arg (no space) |
| `incus storage show` | `remote:pool --target member` | `--target` for member-specific |

### OVN Setup Cheat Sheet

```bash
# 1. Deploy OVN container
incus launch images:debian/12 REMOTE:ovn-central --target NODE
incus exec REMOTE:ovn-central -- apt-get install -y ovn-central ovn-host
incus exec REMOTE:ovn-central -- ovn-nbctl set-connection ptcp:6641:0.0.0.0
incus exec REMOTE:ovn-central -- ovn-sbctl set-connection ptcp:6642:0.0.0.0
incus config device add REMOTE:ovn-central nb-proxy proxy listen=tcp:HOST_IP:6641 connect=tcp:127.0.0.1:6641
incus config device add REMOTE:ovn-central sb-proxy proxy listen=tcp:HOST_IP:6642 connect=tcp:127.0.0.1:6642

# 2. Enable OVN on each IncusOS node
incus query NODE:/os/1.0/services/ovn --request PUT --data '{"config":{"database":"tcp:HOST_IP:6642","enabled":true,"tunnel_address":"NODE_IP","tunnel_protocol":"geneve"},"state":{}}'

# 3. Configure Incus
incus config set REMOTE: network.ovn.northbound_connection tcp:HOST_IP:6641
incus cluster role add REMOTE:MEMBER ovn-chassis  # for each member

# 4. Create UPLINK (per-member then cluster-wide)
incus network create REMOTE:UPLINK --type physical --target MEMBER parent=mgmt  # each member
incus network create REMOTE:UPLINK --type physical ipv4.ovn.ranges=RANGE ipv4.gateway=GW/PREFIX

# 5. Create OVN network
incus network create REMOTE:net-name --type=ovn network=UPLINK ipv4.address=SUBNET ipv4.nat=true
```

### Migration Readiness Checklist

| Setting | Value | Why |
|---------|-------|-----|
| `limits.cpu` | Range (e.g., `0-1`) | Fixed QEMU topology across hosts |
| `migration.stateful` | `true` | Enables live migration |
| `root size.state` | `2GiB` (or `4GiB` for 3-4 vCPUs) | Space for memory state file |

Configure while VM is **stopped**:

```bash
incus stop REMOTE:VM
incus config set REMOTE:VM limits.cpu=0-1
incus config set REMOTE:VM migration.stateful=true
incus config device override REMOTE:VM root size.state=2GiB
incus start REMOTE:VM
```

### Troubleshooting

| Symptom | Cause | Fix |
|---------|-------|-----|
| Port 8443 not reachable after boot | Boot still in progress or crontab bug | Wait 180s; check scrub_schedule via API |
| `scrub_schedule` empty | Crontab race condition | `incusos-proxmox --status` auto-heals |
| `Missing section footer for ICH9LPC` on migration | `limits.cpu` set as integer | Set as range: `limits.cpu=0-1` |
| `VM agent isn't currently running` after migration | Agent reconnecting | `sleep 4` after migration |
| `db.sock not found` on OVN config | OVN service not enabled on IncusOS | Enable via `/os/1.0/services/ovn` API |
| Cross-node ping fails (bridge) | Bridge networks are node-local | Use OVN network instead |
| `zfs load-key: Raw key too short` | TPM corruption from premature VM stop | Destroy and redeploy VM |
| Cluster join fails with "pool already exists" | `apply_defaults: true` on joining node | Use `apply_defaults: false` or run 8-command cleanup |
| OC cannot manage cluster nodes | Nodes deployed with standard ISO | Use OC-provisioned ISO for full integration |
| `CPUID vnmi` warning during migration | Cosmetic QEMU check | Safe to ignore |
| "Parent interface 'ens18' not found" | IncusOS names its NIC `mgmt` | Use `parent=mgmt` for UPLINK network |
| OC CLI "Invalid number of arguments" | OC CLI doesn't support `remote:` suffix | Use `operations-center remote switch NAME` first |
| Container placed on wrong node | Cluster auto-schedules without `--target` | Use `--target NODE` for explicit placement |
| "Device from profile(s) cannot be modified" | `root` device comes from default profile | Use `incus config device override` instead of `device set` |
| `incus launch` hangs or times out | Image download to new node takes 2-3 min | Run launches one at a time, not chained with `&&` |
| Aether CSRF error on API call | Session auth endpoints need CSRF token | Use Playwright or `aether-browser` helper |
| AWX pods OOMKilled | Web pod needs 2 GiB minimum | Check `awx-manifests/base/awx.yaml` resource limits |
| HAProxy 405 on health check | HTTP mode sends OPTIONS requests | Use TCP mode for nginx backends |
| Prometheus scrape failing on Incus | Client certs missing or wrong perms | Check `/etc/prometheus/certs/` in monitoring container |
| Grafana not reachable via forward | Network forward not created | `./deploy-observability --doctor` to diagnose |