maarten
/
incus-contrib
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

OC v0.3.0 comprehensive test: retry logic, auto-heal, apply_defaults validation 

Phase 0-10 of OC test plan completed:
- Add --retries N flag to incusos-proxmox (default 3, configurable boot retries)
- Add fix_scrub_schedule() auto-heal for IncusOS crontab bug via REST API
- Validate apply_defaults: false for OC-managed nodes (recommended approach)
- Test full OC lifecycle: deploy, provision, cluster, workloads, migration,
  evacuation, node failure/recovery, inventory query, REST API
- Update CLAUDE.md: OC v0.3.0 commands, --retries, scrub fix, node recovery
- Update operations-center-guide.md: v0.3.0, apply_defaults, scrub fix,
  node failure simulation, IncusOS REST API docs, inventory query

Key findings:
- apply_defaults: false lets OC Terraform handle everything cleanly
- fix_scrub_schedule() proactively heals ~50% crontab boot failure
- OC does not detect node failures (always shows "ready")
- inventory query (v0.3.0) provides cross-resource tree views
- Node crash recovery: auto-rejoin in ~60s, containers auto-start

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Maarten 2026-02-21 21:41:34 +01:00
parent b8e7a7ca62
commit 52d8dbdc89
5 changed files with 309 additions and 95 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

38
CLAUDE.md
View File

@ -419,8 +419,14 @@ incu-contrib/
as Incus: copy `client.crt` and `client.key` from `~/.config/incus/`).
- **Port**: 8443 (same as Incus on IncusOS) for API, CLI, and web UI.
- **Browser access**: requires PKCS#12 client certificate (`client.pfx`)
imported into the browser.
- **OC is under active development** (v0.2.2). Commands and APIs may change.
imported into the browser. Web UI is a React SPA at `/ui/`.
- **OC is under active development** (v0.3.0). Commands and APIs may change.
Use `operations-center --version` (not `version` subcommand) to check.
- **v0.3.0 new commands**: `provisioning channel` (add/list/show),
`cluster update/update-certificate/rename`, expanded `inventory`
(network-acl, address-sets, load-balancers, peers, zones, integrations,
forwards, storage-buckets, storage-volumes), `inventory query` (cross-resource
tree view with filters and Go templates), `system certificate set`.
- The `--doctor` command on `incusos-proxmox` reports whether the CLI is
installed.
@ -452,10 +458,11 @@ incu-contrib/
```
OC handles: `core.https_address` → cluster enable → joins → seed
application → Terraform config. Adds `meshbr0` network.
- **apply_defaults conflict** (tested): if SEED_DATA has `apply_defaults: true`,
nodes already have storage pool/network/cert. OC's Terraform fails with
"already exists" errors but the cluster forms successfully. Use
`apply_defaults: false` for OC-managed nodes to avoid this.
- **apply_defaults: false is recommended** (tested): use `apply_defaults: false`
for OC-managed nodes. OC's Terraform handles storage pool, network, and cert
creation cleanly. With `apply_defaults: true`, nodes already have these
resources and OC's Terraform fails with "already exists" errors (cluster still
forms, but Terraform artifacts are empty).
#### Tested limitations
@ -468,9 +475,13 @@ incu-contrib/
crash the daemon. See "IncusOS boot failure" section below for full analysis.
Fix: destroy and redeploy OC-managed nodes. Proxmox stop/start is safe.
- **No cluster member state tracking** -- OC always shows `ready` even for
EVACUATED/OFFLINE nodes
EVACUATED/OFFLINE nodes. Does not detect node crashes (Incus heartbeat
detects failure in ~40s; OC has no equivalent).
- **Stale entries** from out-of-band cluster changes persist after resync
- **Server removal blocked** if server is part of an OC cluster
- **Node failure recovery**: Proxmox hard-stop simulates crash. After restart,
node auto-rejoins cluster in ~60s. Containers auto-start. Works cleanly
if crontab bug doesn't hit (auto-healed by `fix_scrub_schedule()`).
- See `notes/operations-center-guide.md` for full tested OC reference.
@ -490,6 +501,12 @@ incu-contrib/
- **`--verbose` / `-v`**: shows detailed output (tool paths, API calls). Default
output is concise (step names + results). `--quiet` suppresses everything
except warnings and errors.
- **`--retries N`**: number of stop+start retries for VMs that fail to boot
(port 8443 not reachable). Default: 3. `--retries 0` disables retries.
Each retry: Proxmox stop → 10s wait → start → 30s wait → poll port 8443.
With ~50% boot failure rate from the crontab bug, 3 retries gives ~93.75%
success rate. After retries, `fix_scrub_schedule()` proactively heals the
root cause via the IncusOS REST API.
### IncusOS first-boot sequence (observed via console screenshots)
@ -627,6 +644,13 @@ from the installed disk by `state.LoadOrCreate()` → `initialize()`.
- **Detection**: `phase_install` checks port 8443 after starting from disk
(30s wait + up to 60s polling). If first check fails, automatic retry
adds ~100s. Failed VMs after retry are reported with remediation.
- **Auto-heal via IncusOS REST API**: `incusos-proxmox` includes
`fix_scrub_schedule()` which proactively fixes empty `scrub_schedule` on
every deployed node via `PUT /os/1.0/system/storage`. The `/os/` prefix
proxies to the IncusOS daemon API through Incus. Safe to call on every
node — returns early if schedule is already set. Called on both initial
success and retry success paths.
- **Filed upstream**: IncusOS issue #843.
- **Source files**: `incus-osd/cmd/incus-osd/main.go` (`registerJobs()`,
`updateChecker()`, `startup()`, `firstBootActions()`),
`incus-osd/internal/scheduling/scheduling.go` (`RegisterJob()`,

1
incusos/README.md
View File

@ -295,6 +295,7 @@ This script ensures all settings are correct, preventing install failures.
-c, --cert FILE Client certificate PEM file (default: auto-detect)
--no-cert Skip certificate injection
--vm NAME Operate on a single VM only
--retries N Boot retries for first-boot failures (default: 3)
-y, --yes Skip confirmation prompts
--proxmox FILE Proxmox connection config (default: auto-discover)
--doctor Check tool versions and environment health

6
incusos/examples/lab-oc-nodes.yaml
View File

@ -23,12 +23,12 @@ defaults:
vms:
- name: oc-node-01
app: incus
apply_defaults: true
apply_defaults: false
- name: oc-node-02
app: incus
apply_defaults: true
apply_defaults: false
- name: oc-node-03
app: incus
apply_defaults: true
apply_defaults: false

169
incusos/incusos-proxmox
View File

@ -135,6 +135,7 @@ ${BOLD}OPTIONS${RESET}
${BOLD}-c, --cert${RESET} FILE Client certificate PEM file
${BOLD}--no-cert${RESET} Skip certificate injection
${BOLD}--vm${RESET} NAME Operate on a single VM only
${BOLD}--retries${RESET} N Boot retries for first-boot failures (default: 3)
${BOLD}-y, --yes${RESET} Skip confirmation prompts
${BOLD}--doctor${RESET} Check tool versions and environment health
${BOLD}--cleanup${RESET} Destroy VMs defined in config (managed only)
@ -304,6 +305,7 @@ LIST_LABS=false
DRY_RUN=false
QUIET=false
VERBOSE=false
INSTALL_RETRIES=3
PROXMOX_CONFIG=""
CONFIG_FILE=""
@ -342,6 +344,14 @@ parse_args() {
YES=true
shift
;;
--retries)
INSTALL_RETRIES="${2:?'--retries requires a number'}"
if ! [[ "$INSTALL_RETRIES" =~ ^[0-9]+$ ]]; then
error "--retries must be a non-negative integer"
exit 1
fi
shift 2
;;
--status)
PHASE="status"
shift
@ -3221,6 +3231,72 @@ register_remote() {
fi
}
# ---------------------------------------------------------------------------
# fix_scrub_schedule -- proactively fix empty scrub_schedule via IncusOS API
# ---------------------------------------------------------------------------
# IncusOS has an intermittent bug where state.txt is written with an empty
# scrub_schedule field (the encoder skips zero values). When registerJobs()
# runs, gocron.IsValid("") fails and the daemon crashes in a loop. The REST
# API is briefly reachable during each cycle (Incus starts before the
# scheduler), so we can fix it via PUT /os/1.0/system/storage.
#
# This function is safe to call on every node -- if scrub_schedule is already
# set correctly, the GET check returns early without making any changes.
#
# Usage: fix_scrub_schedule <ip> <name>
# Returns: 0 on success or not needed, 1 on failure
fix_scrub_schedule() {
local ip="$1"
local name="$2"
# Need client cert for the IncusOS API
if [[ -z "$CERT_FILE_RESOLVED" ]]; then
detail "No client certificate available — skipping scrub_schedule check"
return 0
fi
local cert="$CERT_FILE_RESOLVED"
local key="${CERT_FILE_RESOLVED%.crt}.key"
if [[ ! -f "$key" ]]; then
detail "Client key not found at ${key} — skipping scrub_schedule check"
return 0
fi
# GET current storage config
local response
response=$(curl -sk --connect-timeout 5 --max-time 10 \
--cert "$cert" --key "$key" \
"https://${ip}:8443/os/1.0/system/storage" 2>/dev/null) || {
detail "Could not query IncusOS storage API on '${name}' — skipping scrub_schedule check"
return 0
}
# Check if scrub_schedule is empty or missing
local current_schedule
current_schedule=$(printf '%s' "$response" | \
python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('metadata',{}).get('config',{}).get('scrub_schedule',''))" 2>/dev/null) || current_schedule=""
if [[ -n "$current_schedule" ]]; then
detail "scrub_schedule on '${name}' is already set: ${current_schedule}"
return 0
fi
# scrub_schedule is empty — fix it
local put_response
put_response=$(curl -sk --connect-timeout 5 --max-time 10 \
--cert "$cert" --key "$key" \
-X PUT -H "Content-Type: application/json" \
-d '{"config":{"scrub_schedule":"0 4 * * 0"}}' \
"https://${ip}:8443/os/1.0/system/storage" 2>/dev/null) || {
warn "Failed to fix scrub_schedule on '${name}' via IncusOS API"
return 1
}
info "Fixed empty scrub_schedule on '${name}' via IncusOS API"
return 0
}
# ---------------------------------------------------------------------------
# Phase 5: Install
# ---------------------------------------------------------------------------
@ -3431,53 +3507,66 @@ phase_install() {
echo ""
if [[ "$healthy" == true ]]; then
# Proactively fix empty scrub_schedule (IncusOS crontab bug).
# Safe to call on every node -- returns early if already set.
fix_scrub_schedule "$ip" "$name" || true
success "VM '${name}' installed and running at ${ip}"
register_remote "$name" "$ip"
else
# Retry once: IncusOS has an intermittent "invalid crontab expression"
# Retry loop: IncusOS has an intermittent "invalid crontab expression"
# bug (~50% first-boot hit rate) where registerJobs() fails during
# startup(). Most likely a race condition between the REST API and
# startup sequence, or a gocron library edge case. A stop+start
# cycle resolves it because the transient condition clears on retry.
if [[ -n "$ip" ]]; then
warn "VM '${name}': port 8443 not reachable at ${ip} — retrying (stop+start)"
else
warn "VM '${name}': could not detect IP — retrying (stop+start)"
fi
info "Stopping VM for retry..."
pve_stop_vm "$vmid" || true
sleep 10
info "Starting VM (retry)..."
pve_start_vm "$vmid" || true
sleep 30
# startup(). A stop+start cycle resolves it because the hard power-off
# may discard the corrupt state.txt before it's committed to disk,
# allowing initialize() to recreate it cleanly on next boot.
local retry=0
local retry_ok=false
while [[ $retry -lt $INSTALL_RETRIES ]]; do
retry=$((retry + 1))
if [[ -n "$ip" ]]; then
warn "VM '${name}': port 8443 not reachable at ${ip} — retry ${retry}/${INSTALL_RETRIES} (stop+start)"
else
warn "VM '${name}': could not detect IP — retry ${retry}/${INSTALL_RETRIES} (stop+start)"
fi
info "Stopping VM for retry..."
pve_stop_vm "$vmid" || true
sleep 10
info "Starting VM (retry ${retry}/${INSTALL_RETRIES})..."
pve_start_vm "$vmid" || true
sleep 30
# Re-detect IP (may have changed after restart)
ip=$(pve_vm_ip "$vmid") || ip=""
# Re-detect IP (may have changed after restart)
ip=$(pve_vm_ip "$vmid") || ip=""
local retry_healthy=false
if [[ -n "$ip" ]]; then
local retry_poll=0
while [[ $retry_poll -lt 12 ]]; do
if curl -sk --connect-timeout 3 "https://${ip}:8443" &>/dev/null; then
retry_healthy=true
break
fi
sleep 5
retry_poll=$((retry_poll + 1))
done
fi
if [[ -n "$ip" ]]; then
local retry_poll=0
while [[ $retry_poll -lt 12 ]]; do
if curl -sk --connect-timeout 3 "https://${ip}:8443" &>/dev/null; then
retry_ok=true
break
fi
sleep 5
retry_poll=$((retry_poll + 1))
done
fi
if [[ "$retry_healthy" == true ]]; then
success "VM '${name}' recovered after retry — running at ${ip}"
register_remote "$name" "$ip"
elif [[ -n "$ip" ]]; then
warn "VM '${name}' installed but port 8443 not reachable at ${ip} (even after retry)"
warn "Likely IncusOS boot failure (check Proxmox console for errors)"
warn "Redeploy this VM with: ${SCRIPT_NAME} --cleanup --yes ${CONFIG_FILE} && ${SCRIPT_NAME} --yes ${CONFIG_FILE}"
install_failures=$((install_failures + 1))
else
warn "VM '${name}' installed but could not detect IP (even after retry)"
warn "Check Proxmox console for status"
if [[ "$retry_ok" == true ]]; then
# Proactively fix empty scrub_schedule (IncusOS crontab bug).
fix_scrub_schedule "$ip" "$name" || true
success "VM '${name}' recovered after retry ${retry}/${INSTALL_RETRIES} — running at ${ip}"
register_remote "$name" "$ip"
break
fi
done
if [[ "$retry_ok" != true ]]; then
if [[ -n "$ip" ]]; then
warn "VM '${name}' installed but port 8443 not reachable at ${ip} (after ${INSTALL_RETRIES} retries)"
warn "Likely IncusOS boot failure (check Proxmox console for errors)"
warn "Redeploy this VM with: ${SCRIPT_NAME} --cleanup --yes ${CONFIG_FILE} && ${SCRIPT_NAME} --yes ${CONFIG_FILE}"
else
warn "VM '${name}' installed but could not detect IP (after ${INSTALL_RETRIES} retries)"
warn "Check Proxmox console for status"
fi
install_failures=$((install_failures + 1))
fi
fi

190
notes/operations-center-guide.md
View File

@ -5,7 +5,7 @@ a centralized management layer for IncusOS/Incus deployments. OC handles
provisioning, clustering, inventory, and updates through a REST API, CLI, and
web UI.
All commands tested with Operations Center v0.2.2 and IncusOS build 202602200553
All commands tested with Operations Center v0.3.0 and IncusOS build 202602200553
on Proxmox VE 9.1.5 (nested virtualization on Intel).
---
@ -36,10 +36,10 @@ containing a provisioning token, then self-register on first boot.
```bash
# Check for latest release at https://github.com/FuturFusion/operations-center/releases
wget https://github.com/FuturFusion/operations-center/releases/download/v0.2.2/operations-center_linux_x86_64
wget https://github.com/FuturFusion/operations-center/releases/download/v0.3.0/operations-center_linux_x86_64
chmod +x operations-center_linux_x86_64
sudo mv operations-center_linux_x86_64 /usr/local/bin/operations-center
operations-center version
operations-center --version # Note: use --version, not 'version' subcommand
```
### Set up OC client certificates
@ -233,15 +233,15 @@ defaults:
vms:
- name: oc-node-01
app: incus
apply_defaults: true
apply_defaults: false # OC handles storage/network via Terraform
- name: oc-node-02
app: incus
apply_defaults: true
apply_defaults: false
- name: oc-node-03
app: incus
apply_defaults: true
apply_defaults: false
```
**VMID note**: if your Proxmox API token is scoped to a resource pool, it
@ -316,29 +316,35 @@ OC orchestrates the full cluster formation:
4. Applies the application seed (injects client cert)
5. Runs Terraform/OpenTofu for post-cluster configuration
### apply_defaults conflict (tested finding)
### apply_defaults: false (tested, recommended)
When using `apply_defaults: true` in the SEED_DATA (via `incusos-proxmox`),
each node already has a `local` ZFS storage pool, `incusbr0` network bridge,
and the client certificate. OC's Terraform post-clustering step then tries to
create the same resources and fails with:
**Use `apply_defaults: false`** for OC-managed nodes. This is now the tested
and recommended approach. With `apply_defaults: false`:
- Nodes still listen on port 8443 and trust preseed certificates
- The underlying ZFS dataset (`local/incus`) still exists
- But no Incus storage pool, network, or profile devices are created
- OC's Terraform handles all resource creation cleanly during cluster formation
**Tested result**: cluster forms cleanly with `apply_defaults: false`. OC's
Terraform creates the storage pool (`local`), networks (`incusbr0`, `meshbr0`),
and injects the client certificate via `--application-seed-config`. The only
Terraform error is "Certificate already in trust store" when the cert was also
injected via SEED_DATA -- avoid this by only injecting the cert through the
application seed config, not the SEED_DATA.
### apply_defaults: true (also works, with caveats)
When using `apply_defaults: true`, each node already has a `local` ZFS storage
pool, `incusbr0` network bridge, and the client certificate. OC's Terraform
then fails with:
- "Certificate already in trust store"
- "Network is not in pending state"
- "Storage pool is not in pending state"
**The cluster itself forms successfully** despite these Terraform errors. All
3 nodes join the cluster correctly. The Terraform errors are non-fatal because
the resources already exist. However, the `cluster artifact list` will be empty
since Terraform didn't complete.
**Recommended approaches** for future deployments:
1. **Keep apply_defaults** (current tested approach): cluster forms fine, ignore
Terraform errors. Skip `--application-seed-config` since the cert is already
injected by the seed.
2. **Skip apply_defaults**: deploy nodes without `apply_defaults: true` and
without cert injection in the seed. Let OC handle everything via Terraform.
This is the "clean" approach but hasn't been tested yet.
**The cluster itself forms successfully** despite these Terraform errors.
However, `cluster artifact list` will be empty since Terraform didn't complete.
### Verify cluster (tested)
@ -433,6 +439,59 @@ The proper lifecycle for maintenance is:
4. Wait for IncusOS to boot (~60-90s)
5. `incus cluster restore <remote>:<member> --force`
### IncusOS scrub_schedule fix via API (tested)
IncusOS has an intermittent bug (~50% of deploys) where `state.txt` is written
with an empty `scrub_schedule` field. The `registerJobs()` function calls
`gocron.IsValid("")` which fails, crashing the daemon in a loop. The REST API
is briefly reachable during each crash cycle (Incus starts before the scheduler).
**Fix via IncusOS REST API** (through the Incus proxy at `/os/` prefix):
```bash
# Check current storage config
curl -sk --cert ~/.config/incus/client.crt \
--key ~/.config/incus/client.key \
https://<NODE_IP>:8443/os/1.0/system/storage
# If scrub_schedule is empty, fix it
curl -sk --cert ~/.config/incus/client.crt \
--key ~/.config/incus/client.key \
-X PUT -H "Content-Type: application/json" \
-d '{"config":{"scrub_schedule":"0 4 * * 0"}}' \
https://<NODE_IP>:8443/os/1.0/system/storage
```
**Key details**:
- The `/os/` prefix is required — this proxies to the IncusOS daemon API
- PATCH returns 501 (not implemented); use PUT instead
- `incus admin os system storage edit` fails because it reads the current
(corrupt) config and validates it before allowing edits
- `incusos-proxmox` now includes `fix_scrub_schedule()` which proactively
fixes this on every deployed node (safe — returns early if already set)
- Filed upstream as IncusOS issue #843
### Node failure and recovery (tested)
Proxmox hard-stop simulates node crash. Incus cluster detects failure via
heartbeat timeout (~30-40s). Node auto-rejoins after Proxmox restart (~60s boot):
```bash
# Simulate crash: Proxmox API stop
curl -sk -X POST ".../nodes/pve/qemu/<VMID>/status/stop" ...
# After ~40s: incus cluster list shows OFFLINE with heartbeat info
# Restart: Proxmox API start
curl -sk -X POST ".../nodes/pve/qemu/<VMID>/status/start" ...
# After ~60s boot: node auto-rejoins cluster (ONLINE)
# Containers on the node auto-start after recovery
```
**OC does not detect node failures**: `provisioning server show` still shows
`Status: ready` for an OFFLINE/crashed node. `Last Seen` timestamp is stale.
**OC does not track Incus cluster member state**: the server list always
shows `ready` status even when the Incus cluster reports `EVACUATED`.
@ -542,6 +601,7 @@ printf "yes\n" | incus cluster remove oc-node-01:oc-node-03 --force
# 2. Destroy broken VM via Proxmox (stop + delete)
# 3. Redeploy via incusos-proxmox --iso (single-node config)
# Use apply_defaults: false for OC-managed nodes
./incusos/incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes /tmp/redeploy-node-03.yaml
# 4. New node auto-registers with OC (same hostname, new IP)
@ -550,18 +610,10 @@ printf "yes\n" | incus cluster remove oc-node-01:oc-node-03 --force
# 5. Manual cluster join (since node was force-removed)
incus config set oc-node-03: core.https_address <NEW_IP>:8443
# Remove apply_defaults resources
incus config unset oc-node-03: storage.backups_volume
incus config unset oc-node-03: storage.images_volume
incus storage volume delete oc-node-03:local backups
incus storage volume delete oc-node-03:local images
incus profile device remove oc-node-03:default root
incus profile device remove oc-node-03:default eth0
incus storage delete oc-node-03:local
incus network delete oc-node-03:incusbr0
# With apply_defaults: false, no cleanup needed (no pool/network to delete)
# Generate token and join
incus cluster add oc-node-01:oc-node-03
printf '\n\nyes\n\nlocal/incus\nlocal/incus\n' | incus cluster join oc-node-01: oc-node-03:
printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join oc-node-01: oc-node-03:
# Note: extra \n for meshbr0 tunnel.mesh.interface prompt (added by OC clustering)
# 6. Fix remote after cert change
@ -569,6 +621,21 @@ incus remote remove oc-node-03
incus remote add oc-node-03 https://<NEW_IP>:8443 --accept-certificate
```
### Node failure simulation (tested)
Proxmox hard-stop simulates node crash. Incus detects via heartbeat (~40s):
```bash
# Simulate crash
curl -sk -X POST ".../nodes/pve/qemu/<VMID>/status/stop" ...
# incus cluster list shows OFFLINE after ~40s
# Restart
curl -sk -X POST ".../nodes/pve/qemu/<VMID>/status/start" ...
# Node auto-rejoins after ~60s boot. Containers auto-start.
# OC still shows "ready" (does not detect node failures)
```
### Factory reset (not tested)
```bash
@ -650,10 +717,20 @@ operations-center provisioning cluster factory-reset oc-cluster <token-uuid>
| `inventory instance list` | List all instances |
| `inventory instance show <UUID>` | Instance details (by UUID, not name) |
| `inventory storage-pool list` | List storage pools |
| `inventory storage-volume list` | List storage volumes |
| `inventory storage-bucket list` | List storage buckets |
| `inventory network list` | List networks |
| `inventory network-acl list` | List network ACLs |
| `inventory network-address-set list` | List network address sets |
| `inventory network-forward list` | List network forwards |
| `inventory network-integration list` | List network integrations |
| `inventory network-load-balancer list` | List network load balancers |
| `inventory network-peer list` | List network peers |
| `inventory network-zone list` | List network zones |
| `inventory profile list` | List profiles |
| `inventory project list` | List projects |
| `inventory image list` | List images |
| `inventory query --cluster <name>` | Cross-resource tree view (v0.3.0+) |
### Updates
@ -683,6 +760,8 @@ operations-center provisioning cluster factory-reset oc-cluster <token-uuid>
| `admin os service list` | List services (iscsi, lvm, nvme, ovn, tailscale, ...) |
| `admin os service show <name>` | Service config |
| `admin os debug log` | Debug logs |
| `admin os debug processes` | System process list |
| `admin os debug secureboot` | Secure Boot debug info |
All commands prefixed with `operations-center`.
@ -713,6 +792,19 @@ curl -sk --cert ~/.config/incus/client.crt \
| GET | `/1.0/provisioning/clusters` | List clusters |
| POST | `/1.0/provisioning/clusters` | Form cluster |
| GET | `/1.0/inventory/instances` | List all instances |
| GET | `/1.0/inventory/instance/<UUID>` | Instance details |
| GET | `/1.0/provisioning/servers/<name>` | Server details + hardware inventory |
### IncusOS REST API (via Incus proxy)
IncusOS nodes expose their own REST API through the Incus proxy at the `/os/`
prefix on port 8443:
| Method | Endpoint | Description |
|--------|----------|-------------|
| GET | `/os/1.0/system/storage` | Storage config (scrub_schedule) |
| PUT | `/os/1.0/system/storage` | Update storage config |
| GET | `/os/1.0` | IncusOS system info |
---
@ -771,7 +863,7 @@ ip neigh show | grep -i "$MAC" | awk '{print $1}'
## Known Limitations (Tested Findings)
### Confirmed limitations (v0.2.2)
### Confirmed limitations (v0.3.0)
- **No brownfield adoption**: nodes must boot from OC-provisioned ISO
- **No real-time inventory sync**: requires explicit `cluster resync`
@ -869,20 +961,24 @@ ip neigh show | grep -i "$MAC" | awk '{print $1}'
## Open Research Items
These are observed behaviors whose root cause is not yet confirmed:
- **OC reboot failure mechanism** (resolved): guest reboot is safe on
standalone IncusOS nodes (tested: simultaneous 3-node reboot, all recover
in ~50s). The failure is OC-specific: the OC agent runs on every boot and
fails with configuration errors (invalid crontab), re-registration
conflicts (constraint violation), or cluster quorum loss (peers in boot
loops). The root cause is NOT dqlite corruption or TPM issues.
in ~50s). The failure is OC-specific: the OC agent pushes config via the
IncusOS REST API that gets persisted to `state.txt`. On reboot, invalid
values (e.g., cron expression where Go duration is expected) crash the
daemon. Root cause is NOT dqlite corruption or TPM issues.
- **apply_defaults conflict with OC Terraform**: when nodes have
`apply_defaults: true`, OC's post-clustering Terraform step fails with
"already in trust store" / "not in pending state" errors. The cluster forms
successfully despite these errors. Unclear whether skipping `apply_defaults`
would allow clean Terraform completion (untested).
- **apply_defaults conflict with OC Terraform** (resolved): `apply_defaults: false`
is now the tested and recommended approach. OC's Terraform handles storage pool,
network, and cert creation cleanly when nodes start without defaults.
`apply_defaults: true` also works but produces non-fatal Terraform errors.
- **IncusOS scrub_schedule bug** (workaround found, upstream filed as #843):
~50% of deploys hit an empty `scrub_schedule` in `state.txt`. Root cause is
likely a race condition between REST API startup and `registerJobs()`, or a
zero-value encoding bug in `encodeHelper()`. Workaround: `PUT /os/1.0/system/storage`
with `{"config":{"scrub_schedule":"0 4 * * 0"}}`. `incusos-proxmox` now
auto-heals this proactively on every deploy.
- **Stale server entries after out-of-band changes**: when cluster members are
removed via `incus cluster remove` (not through OC), stale entries persist in
@ -893,6 +989,10 @@ These are observed behaviors whose root cause is not yet confirmed:
to pick up changes made via the `incus` CLI. No webhook or event-driven sync.
This applies to instance creation, migration, evacuation, and deletion.
- **Node failure detection**: OC does not detect node failures. `provisioning
server show` continues to report `Status: ready` for OFFLINE/crashed nodes.
Incus cluster detects failures via heartbeat (~40s), but OC has no equivalent.
---
## References