---
paths:
  - "bin/lab-test"
  - "bin/incusos-proxmox"
  - "notes/clustering-guide.md"
  - "notes/production-lab-guide.md"
---

# Incus Clustering

## Overview

- Cluster formation via `incus` CLI using remotes. No SSH needed (IncusOS is immutable).
- No VIP needed: each node advertises its own IP. Requests forwarded internally.

## Pre-clustering: fix core.https_address

- Default `:8443` (wildcard). Clustering needs specific routable IP.
- Set on every node BEFORE clustering:
  ```bash
  incus config set <remote>: core.https_address <NODE_IP>:8443
  ```
- Get routable IP:
  ```bash
  incus query <remote>:/1.0 | python3 -c "import sys,json; d=json.load(sys.stdin); \
    [print(a) for a in d['environment']['addresses'] \
     if not a.startswith('10.') and not a.startswith('fd42:') and not a.startswith('[')]"
  ```

## Cluster enable (init node)

- ```bash
  incus cluster enable <remote>: <member-name>
  ```
  TWO arguments: `<remote>:` (trailing colon) and `<member-name>`.
- Generates new TLS cert. Fix remote:
  ```bash
  incus remote switch local
  incus remote remove <remote>
  incus remote add <remote> https://<NODE_IP>:8443 --accept-certificate
  ```

## apply_defaults for clusters

- **Bootstrap node**: `apply_defaults: true` (needs pool + network)
- **Joining nodes**: `apply_defaults: false` (recommended, clean join)
- **Standalone**: `apply_defaults: true`
- If `apply_defaults: true` on joiners, 8-command cleanup needed before join
  (delete pool, network, volumes, profile devices).

## Join workflow

- **Token**: `incus cluster add <init-remote>:<new-member-name>` (ONE arg)
- **Join**: `incus cluster join <init-remote>: <joining-remote>:` (TWO args)
- **Automated**:
  ```bash
  printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join <init-remote>: <joining-remote>:
  ```
- After join: fix remote (same as init).

## Command syntax gotchas

- `cluster enable remote: member-name` -- TWO args
- `cluster add remote:member-name` -- ONE arg
- `cluster remove remote:member-name --force` -- ONE arg, prompts "yes/no"
- `cluster evacuate remote:member-name` -- ONE arg
- `cluster restore remote:member-name` -- ONE arg
- `cluster join init-remote: joining-remote:` -- TWO args
- `config set remote: key value` -- remote with trailing colon + space
- General: `remote:resource` for resource, `remote:` for server itself

## Workload placement and migration

- **Targeted launch**: `incus launch images:debian/12 <remote>:name --target <member>`
- **Container migration**: stop/move/start only (no CRIU).
- **VM live migration**: requires `migration.stateful=true` (set while stopped).
- **limits.cpu MUST be a range** (e.g., `0-1`), not integer. Integer causes
  `maxcpus` mismatch -> `Missing section footer for ICH9LPC`.
- **size.state required**: `size.state=2GiB` on root disk for stateful ops.
- **Agent reconnect**: sleep 4s after migration before `incus exec`.
- **Evacuation**: `incus cluster evacuate <remote>:<member> --force`
  Use `--action stop` without limits.cpu range fix.

## Cluster rebalancing

```bash
incus config set <remote>: cluster.rebalance.interval=1
incus config set <remote>: cluster.rebalance.threshold=10
incus config set <remote>: cluster.rebalance.batch=2
incus config set <remote>: cluster.rebalance.cooldown=5m
```

## Lab validation (lab-test)

- Reads same YAML config as `incusos-proxmox`.
- Phases: deploy, single, cluster, workload, migrate.
- Test instances: `test-*` prefix. Reports PASS/FAIL/SKIP.