incus-contrib/.claude/rules/clustering.md

paths
bin/lab-test bin/incusos-proxmox notes/clustering-guide.md notes/production-lab-guide.md

Incus Clustering

Overview

• Cluster formation via incus CLI using remotes. No SSH needed (IncusOS is immutable).
• No VIP needed: each node advertises its own IP. Requests forwarded internally.

Pre-clustering: fix core.https_address

• Default :8443 (wildcard). Clustering needs specific routable IP.
• Set on every node BEFORE clustering:

  incus config set <remote>: core.https_address <NODE_IP>:8443
  

• Get routable IP:

  incus query <remote>:/1.0 | python3 -c "import sys,json; d=json.load(sys.stdin); \
    [print(a) for a in d['environment']['addresses'] \
     if not a.startswith('10.') and not a.startswith('fd42:') and not a.startswith('[')]"
  

Cluster enable (init node)

• incus cluster enable <remote>: <member-name>
  

  TWO arguments: <remote>: (trailing colon) and <member-name>.
• Generates new TLS cert. Fix remote:

  incus remote switch local
  incus remote remove <remote>
  incus remote add <remote> https://<NODE_IP>:8443 --accept-certificate
  

apply_defaults for clusters

• Bootstrap node: apply_defaults: true (needs pool + network)
• Joining nodes: apply_defaults: false (recommended, clean join)
• Standalone: apply_defaults: true
• If apply_defaults: true on joiners, 8-command cleanup needed before join (delete pool, network, volumes, profile devices).

Join workflow

• Token: incus cluster add <init-remote>:<new-member-name> (ONE arg)
• Join: incus cluster join <init-remote>: <joining-remote>: (TWO args)
• Automated:

  printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join <init-remote>: <joining-remote>:
  

• After join: fix remote (same as init).

Command syntax gotchas

• cluster enable remote: member-name -- TWO args
• cluster add remote:member-name -- ONE arg
• cluster remove remote:member-name --force -- ONE arg, prompts "yes/no"
• cluster evacuate remote:member-name -- ONE arg
• cluster restore remote:member-name -- ONE arg
• cluster join init-remote: joining-remote: -- TWO args
• config set remote: key value -- remote with trailing colon + space
• General: remote:resource for resource, remote: for server itself

Workload placement and migration

• Targeted launch: incus launch images:debian/12 <remote>:name --target <member>
• Container migration: stop/move/start only (no CRIU).
• VM live migration: requires migration.stateful=true (set while stopped).
• limits.cpu MUST be a range (e.g., 0-1), not integer. Integer causes maxcpus mismatch -> Missing section footer for ICH9LPC.
• size.state required: size.state=2GiB on root disk for stateful ops.
• Agent reconnect: sleep 4s after migration before incus exec.
• Evacuation: incus cluster evacuate <remote>:<member> --force Use --action stop without limits.cpu range fix.

Cluster rebalancing

incus config set <remote>: cluster.rebalance.interval=1
incus config set <remote>: cluster.rebalance.threshold=10
incus config set <remote>: cluster.rebalance.batch=2
incus config set <remote>: cluster.rebalance.cooldown=5m

Lab validation (lab-test)

• Reads same YAML config as incusos-proxmox.
• Phases: deploy, single, cluster, workload, migrate.
• Test instances: test-* prefix. Reports PASS/FAIL/SKIP.