OC v0.3.0 comprehensive test: retry logic, auto-heal, apply_defaults validation
Phase 0-10 of OC test plan completed: - Add --retries N flag to incusos-proxmox (default 3, configurable boot retries) - Add fix_scrub_schedule() auto-heal for IncusOS crontab bug via REST API - Validate apply_defaults: false for OC-managed nodes (recommended approach) - Test full OC lifecycle: deploy, provision, cluster, workloads, migration, evacuation, node failure/recovery, inventory query, REST API - Update CLAUDE.md: OC v0.3.0 commands, --retries, scrub fix, node recovery - Update operations-center-guide.md: v0.3.0, apply_defaults, scrub fix, node failure simulation, IncusOS REST API docs, inventory query Key findings: - apply_defaults: false lets OC Terraform handle everything cleanly - fix_scrub_schedule() proactively heals ~50% crontab boot failure - OC does not detect node failures (always shows "ready") - inventory query (v0.3.0) provides cross-resource tree views - Node crash recovery: auto-rejoin in ~60s, containers auto-start Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
b8e7a7ca62
commit
52d8dbdc89
38
CLAUDE.md
38
CLAUDE.md
|
|
@ -419,8 +419,14 @@ incu-contrib/
|
|||
as Incus: copy `client.crt` and `client.key` from `~/.config/incus/`).
|
||||
- **Port**: 8443 (same as Incus on IncusOS) for API, CLI, and web UI.
|
||||
- **Browser access**: requires PKCS#12 client certificate (`client.pfx`)
|
||||
imported into the browser.
|
||||
- **OC is under active development** (v0.2.2). Commands and APIs may change.
|
||||
imported into the browser. Web UI is a React SPA at `/ui/`.
|
||||
- **OC is under active development** (v0.3.0). Commands and APIs may change.
|
||||
Use `operations-center --version` (not `version` subcommand) to check.
|
||||
- **v0.3.0 new commands**: `provisioning channel` (add/list/show),
|
||||
`cluster update/update-certificate/rename`, expanded `inventory`
|
||||
(network-acl, address-sets, load-balancers, peers, zones, integrations,
|
||||
forwards, storage-buckets, storage-volumes), `inventory query` (cross-resource
|
||||
tree view with filters and Go templates), `system certificate set`.
|
||||
- The `--doctor` command on `incusos-proxmox` reports whether the CLI is
|
||||
installed.
|
||||
|
||||
|
|
@ -452,10 +458,11 @@ incu-contrib/
|
|||
```
|
||||
OC handles: `core.https_address` → cluster enable → joins → seed
|
||||
application → Terraform config. Adds `meshbr0` network.
|
||||
- **apply_defaults conflict** (tested): if SEED_DATA has `apply_defaults: true`,
|
||||
nodes already have storage pool/network/cert. OC's Terraform fails with
|
||||
"already exists" errors but the cluster forms successfully. Use
|
||||
`apply_defaults: false` for OC-managed nodes to avoid this.
|
||||
- **apply_defaults: false is recommended** (tested): use `apply_defaults: false`
|
||||
for OC-managed nodes. OC's Terraform handles storage pool, network, and cert
|
||||
creation cleanly. With `apply_defaults: true`, nodes already have these
|
||||
resources and OC's Terraform fails with "already exists" errors (cluster still
|
||||
forms, but Terraform artifacts are empty).
|
||||
|
||||
#### Tested limitations
|
||||
|
||||
|
|
@ -468,9 +475,13 @@ incu-contrib/
|
|||
crash the daemon. See "IncusOS boot failure" section below for full analysis.
|
||||
Fix: destroy and redeploy OC-managed nodes. Proxmox stop/start is safe.
|
||||
- **No cluster member state tracking** -- OC always shows `ready` even for
|
||||
EVACUATED/OFFLINE nodes
|
||||
EVACUATED/OFFLINE nodes. Does not detect node crashes (Incus heartbeat
|
||||
detects failure in ~40s; OC has no equivalent).
|
||||
- **Stale entries** from out-of-band cluster changes persist after resync
|
||||
- **Server removal blocked** if server is part of an OC cluster
|
||||
- **Node failure recovery**: Proxmox hard-stop simulates crash. After restart,
|
||||
node auto-rejoins cluster in ~60s. Containers auto-start. Works cleanly
|
||||
if crontab bug doesn't hit (auto-healed by `fix_scrub_schedule()`).
|
||||
|
||||
- See `notes/operations-center-guide.md` for full tested OC reference.
|
||||
|
||||
|
|
@ -490,6 +501,12 @@ incu-contrib/
|
|||
- **`--verbose` / `-v`**: shows detailed output (tool paths, API calls). Default
|
||||
output is concise (step names + results). `--quiet` suppresses everything
|
||||
except warnings and errors.
|
||||
- **`--retries N`**: number of stop+start retries for VMs that fail to boot
|
||||
(port 8443 not reachable). Default: 3. `--retries 0` disables retries.
|
||||
Each retry: Proxmox stop → 10s wait → start → 30s wait → poll port 8443.
|
||||
With ~50% boot failure rate from the crontab bug, 3 retries gives ~93.75%
|
||||
success rate. After retries, `fix_scrub_schedule()` proactively heals the
|
||||
root cause via the IncusOS REST API.
|
||||
|
||||
### IncusOS first-boot sequence (observed via console screenshots)
|
||||
|
||||
|
|
@ -627,6 +644,13 @@ from the installed disk by `state.LoadOrCreate()` → `initialize()`.
|
|||
- **Detection**: `phase_install` checks port 8443 after starting from disk
|
||||
(30s wait + up to 60s polling). If first check fails, automatic retry
|
||||
adds ~100s. Failed VMs after retry are reported with remediation.
|
||||
- **Auto-heal via IncusOS REST API**: `incusos-proxmox` includes
|
||||
`fix_scrub_schedule()` which proactively fixes empty `scrub_schedule` on
|
||||
every deployed node via `PUT /os/1.0/system/storage`. The `/os/` prefix
|
||||
proxies to the IncusOS daemon API through Incus. Safe to call on every
|
||||
node — returns early if schedule is already set. Called on both initial
|
||||
success and retry success paths.
|
||||
- **Filed upstream**: IncusOS issue #843.
|
||||
- **Source files**: `incus-osd/cmd/incus-osd/main.go` (`registerJobs()`,
|
||||
`updateChecker()`, `startup()`, `firstBootActions()`),
|
||||
`incus-osd/internal/scheduling/scheduling.go` (`RegisterJob()`,
|
||||
|
|
|
|||
|
|
@ -295,6 +295,7 @@ This script ensures all settings are correct, preventing install failures.
|
|||
-c, --cert FILE Client certificate PEM file (default: auto-detect)
|
||||
--no-cert Skip certificate injection
|
||||
--vm NAME Operate on a single VM only
|
||||
--retries N Boot retries for first-boot failures (default: 3)
|
||||
-y, --yes Skip confirmation prompts
|
||||
--proxmox FILE Proxmox connection config (default: auto-discover)
|
||||
--doctor Check tool versions and environment health
|
||||
|
|
|
|||
|
|
@ -23,12 +23,12 @@ defaults:
|
|||
vms:
|
||||
- name: oc-node-01
|
||||
app: incus
|
||||
apply_defaults: true
|
||||
apply_defaults: false
|
||||
|
||||
- name: oc-node-02
|
||||
app: incus
|
||||
apply_defaults: true
|
||||
apply_defaults: false
|
||||
|
||||
- name: oc-node-03
|
||||
app: incus
|
||||
apply_defaults: true
|
||||
apply_defaults: false
|
||||
|
|
|
|||
|
|
@ -135,6 +135,7 @@ ${BOLD}OPTIONS${RESET}
|
|||
${BOLD}-c, --cert${RESET} FILE Client certificate PEM file
|
||||
${BOLD}--no-cert${RESET} Skip certificate injection
|
||||
${BOLD}--vm${RESET} NAME Operate on a single VM only
|
||||
${BOLD}--retries${RESET} N Boot retries for first-boot failures (default: 3)
|
||||
${BOLD}-y, --yes${RESET} Skip confirmation prompts
|
||||
${BOLD}--doctor${RESET} Check tool versions and environment health
|
||||
${BOLD}--cleanup${RESET} Destroy VMs defined in config (managed only)
|
||||
|
|
@ -304,6 +305,7 @@ LIST_LABS=false
|
|||
DRY_RUN=false
|
||||
QUIET=false
|
||||
VERBOSE=false
|
||||
INSTALL_RETRIES=3
|
||||
PROXMOX_CONFIG=""
|
||||
CONFIG_FILE=""
|
||||
|
||||
|
|
@ -342,6 +344,14 @@ parse_args() {
|
|||
YES=true
|
||||
shift
|
||||
;;
|
||||
--retries)
|
||||
INSTALL_RETRIES="${2:?'--retries requires a number'}"
|
||||
if ! [[ "$INSTALL_RETRIES" =~ ^[0-9]+$ ]]; then
|
||||
error "--retries must be a non-negative integer"
|
||||
exit 1
|
||||
fi
|
||||
shift 2
|
||||
;;
|
||||
--status)
|
||||
PHASE="status"
|
||||
shift
|
||||
|
|
@ -3221,6 +3231,72 @@ register_remote() {
|
|||
fi
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# fix_scrub_schedule -- proactively fix empty scrub_schedule via IncusOS API
|
||||
# ---------------------------------------------------------------------------
|
||||
# IncusOS has an intermittent bug where state.txt is written with an empty
|
||||
# scrub_schedule field (the encoder skips zero values). When registerJobs()
|
||||
# runs, gocron.IsValid("") fails and the daemon crashes in a loop. The REST
|
||||
# API is briefly reachable during each cycle (Incus starts before the
|
||||
# scheduler), so we can fix it via PUT /os/1.0/system/storage.
|
||||
#
|
||||
# This function is safe to call on every node -- if scrub_schedule is already
|
||||
# set correctly, the GET check returns early without making any changes.
|
||||
#
|
||||
# Usage: fix_scrub_schedule <ip> <name>
|
||||
# Returns: 0 on success or not needed, 1 on failure
|
||||
fix_scrub_schedule() {
|
||||
local ip="$1"
|
||||
local name="$2"
|
||||
|
||||
# Need client cert for the IncusOS API
|
||||
if [[ -z "$CERT_FILE_RESOLVED" ]]; then
|
||||
detail "No client certificate available — skipping scrub_schedule check"
|
||||
return 0
|
||||
fi
|
||||
|
||||
local cert="$CERT_FILE_RESOLVED"
|
||||
local key="${CERT_FILE_RESOLVED%.crt}.key"
|
||||
|
||||
if [[ ! -f "$key" ]]; then
|
||||
detail "Client key not found at ${key} — skipping scrub_schedule check"
|
||||
return 0
|
||||
fi
|
||||
|
||||
# GET current storage config
|
||||
local response
|
||||
response=$(curl -sk --connect-timeout 5 --max-time 10 \
|
||||
--cert "$cert" --key "$key" \
|
||||
"https://${ip}:8443/os/1.0/system/storage" 2>/dev/null) || {
|
||||
detail "Could not query IncusOS storage API on '${name}' — skipping scrub_schedule check"
|
||||
return 0
|
||||
}
|
||||
|
||||
# Check if scrub_schedule is empty or missing
|
||||
local current_schedule
|
||||
current_schedule=$(printf '%s' "$response" | \
|
||||
python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('metadata',{}).get('config',{}).get('scrub_schedule',''))" 2>/dev/null) || current_schedule=""
|
||||
|
||||
if [[ -n "$current_schedule" ]]; then
|
||||
detail "scrub_schedule on '${name}' is already set: ${current_schedule}"
|
||||
return 0
|
||||
fi
|
||||
|
||||
# scrub_schedule is empty — fix it
|
||||
local put_response
|
||||
put_response=$(curl -sk --connect-timeout 5 --max-time 10 \
|
||||
--cert "$cert" --key "$key" \
|
||||
-X PUT -H "Content-Type: application/json" \
|
||||
-d '{"config":{"scrub_schedule":"0 4 * * 0"}}' \
|
||||
"https://${ip}:8443/os/1.0/system/storage" 2>/dev/null) || {
|
||||
warn "Failed to fix scrub_schedule on '${name}' via IncusOS API"
|
||||
return 1
|
||||
}
|
||||
|
||||
info "Fixed empty scrub_schedule on '${name}' via IncusOS API"
|
||||
return 0
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Phase 5: Install
|
||||
# ---------------------------------------------------------------------------
|
||||
|
|
@ -3431,35 +3507,41 @@ phase_install() {
|
|||
|
||||
echo ""
|
||||
if [[ "$healthy" == true ]]; then
|
||||
# Proactively fix empty scrub_schedule (IncusOS crontab bug).
|
||||
# Safe to call on every node -- returns early if already set.
|
||||
fix_scrub_schedule "$ip" "$name" || true
|
||||
success "VM '${name}' installed and running at ${ip}"
|
||||
register_remote "$name" "$ip"
|
||||
else
|
||||
# Retry once: IncusOS has an intermittent "invalid crontab expression"
|
||||
# Retry loop: IncusOS has an intermittent "invalid crontab expression"
|
||||
# bug (~50% first-boot hit rate) where registerJobs() fails during
|
||||
# startup(). Most likely a race condition between the REST API and
|
||||
# startup sequence, or a gocron library edge case. A stop+start
|
||||
# cycle resolves it because the transient condition clears on retry.
|
||||
# startup(). A stop+start cycle resolves it because the hard power-off
|
||||
# may discard the corrupt state.txt before it's committed to disk,
|
||||
# allowing initialize() to recreate it cleanly on next boot.
|
||||
local retry=0
|
||||
local retry_ok=false
|
||||
while [[ $retry -lt $INSTALL_RETRIES ]]; do
|
||||
retry=$((retry + 1))
|
||||
if [[ -n "$ip" ]]; then
|
||||
warn "VM '${name}': port 8443 not reachable at ${ip} — retrying (stop+start)"
|
||||
warn "VM '${name}': port 8443 not reachable at ${ip} — retry ${retry}/${INSTALL_RETRIES} (stop+start)"
|
||||
else
|
||||
warn "VM '${name}': could not detect IP — retrying (stop+start)"
|
||||
warn "VM '${name}': could not detect IP — retry ${retry}/${INSTALL_RETRIES} (stop+start)"
|
||||
fi
|
||||
info "Stopping VM for retry..."
|
||||
pve_stop_vm "$vmid" || true
|
||||
sleep 10
|
||||
info "Starting VM (retry)..."
|
||||
info "Starting VM (retry ${retry}/${INSTALL_RETRIES})..."
|
||||
pve_start_vm "$vmid" || true
|
||||
sleep 30
|
||||
|
||||
# Re-detect IP (may have changed after restart)
|
||||
ip=$(pve_vm_ip "$vmid") || ip=""
|
||||
|
||||
local retry_healthy=false
|
||||
if [[ -n "$ip" ]]; then
|
||||
local retry_poll=0
|
||||
while [[ $retry_poll -lt 12 ]]; do
|
||||
if curl -sk --connect-timeout 3 "https://${ip}:8443" &>/dev/null; then
|
||||
retry_healthy=true
|
||||
retry_ok=true
|
||||
break
|
||||
fi
|
||||
sleep 5
|
||||
|
|
@ -3467,17 +3549,24 @@ phase_install() {
|
|||
done
|
||||
fi
|
||||
|
||||
if [[ "$retry_healthy" == true ]]; then
|
||||
success "VM '${name}' recovered after retry — running at ${ip}"
|
||||
if [[ "$retry_ok" == true ]]; then
|
||||
# Proactively fix empty scrub_schedule (IncusOS crontab bug).
|
||||
fix_scrub_schedule "$ip" "$name" || true
|
||||
success "VM '${name}' recovered after retry ${retry}/${INSTALL_RETRIES} — running at ${ip}"
|
||||
register_remote "$name" "$ip"
|
||||
elif [[ -n "$ip" ]]; then
|
||||
warn "VM '${name}' installed but port 8443 not reachable at ${ip} (even after retry)"
|
||||
break
|
||||
fi
|
||||
done
|
||||
|
||||
if [[ "$retry_ok" != true ]]; then
|
||||
if [[ -n "$ip" ]]; then
|
||||
warn "VM '${name}' installed but port 8443 not reachable at ${ip} (after ${INSTALL_RETRIES} retries)"
|
||||
warn "Likely IncusOS boot failure (check Proxmox console for errors)"
|
||||
warn "Redeploy this VM with: ${SCRIPT_NAME} --cleanup --yes ${CONFIG_FILE} && ${SCRIPT_NAME} --yes ${CONFIG_FILE}"
|
||||
install_failures=$((install_failures + 1))
|
||||
else
|
||||
warn "VM '${name}' installed but could not detect IP (even after retry)"
|
||||
warn "VM '${name}' installed but could not detect IP (after ${INSTALL_RETRIES} retries)"
|
||||
warn "Check Proxmox console for status"
|
||||
fi
|
||||
install_failures=$((install_failures + 1))
|
||||
fi
|
||||
fi
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@ a centralized management layer for IncusOS/Incus deployments. OC handles
|
|||
provisioning, clustering, inventory, and updates through a REST API, CLI, and
|
||||
web UI.
|
||||
|
||||
All commands tested with Operations Center v0.2.2 and IncusOS build 202602200553
|
||||
All commands tested with Operations Center v0.3.0 and IncusOS build 202602200553
|
||||
on Proxmox VE 9.1.5 (nested virtualization on Intel).
|
||||
|
||||
---
|
||||
|
|
@ -36,10 +36,10 @@ containing a provisioning token, then self-register on first boot.
|
|||
|
||||
```bash
|
||||
# Check for latest release at https://github.com/FuturFusion/operations-center/releases
|
||||
wget https://github.com/FuturFusion/operations-center/releases/download/v0.2.2/operations-center_linux_x86_64
|
||||
wget https://github.com/FuturFusion/operations-center/releases/download/v0.3.0/operations-center_linux_x86_64
|
||||
chmod +x operations-center_linux_x86_64
|
||||
sudo mv operations-center_linux_x86_64 /usr/local/bin/operations-center
|
||||
operations-center version
|
||||
operations-center --version # Note: use --version, not 'version' subcommand
|
||||
```
|
||||
|
||||
### Set up OC client certificates
|
||||
|
|
@ -233,15 +233,15 @@ defaults:
|
|||
vms:
|
||||
- name: oc-node-01
|
||||
app: incus
|
||||
apply_defaults: true
|
||||
apply_defaults: false # OC handles storage/network via Terraform
|
||||
|
||||
- name: oc-node-02
|
||||
app: incus
|
||||
apply_defaults: true
|
||||
apply_defaults: false
|
||||
|
||||
- name: oc-node-03
|
||||
app: incus
|
||||
apply_defaults: true
|
||||
apply_defaults: false
|
||||
```
|
||||
|
||||
**VMID note**: if your Proxmox API token is scoped to a resource pool, it
|
||||
|
|
@ -316,29 +316,35 @@ OC orchestrates the full cluster formation:
|
|||
4. Applies the application seed (injects client cert)
|
||||
5. Runs Terraform/OpenTofu for post-cluster configuration
|
||||
|
||||
### apply_defaults conflict (tested finding)
|
||||
### apply_defaults: false (tested, recommended)
|
||||
|
||||
When using `apply_defaults: true` in the SEED_DATA (via `incusos-proxmox`),
|
||||
each node already has a `local` ZFS storage pool, `incusbr0` network bridge,
|
||||
and the client certificate. OC's Terraform post-clustering step then tries to
|
||||
create the same resources and fails with:
|
||||
**Use `apply_defaults: false`** for OC-managed nodes. This is now the tested
|
||||
and recommended approach. With `apply_defaults: false`:
|
||||
|
||||
- Nodes still listen on port 8443 and trust preseed certificates
|
||||
- The underlying ZFS dataset (`local/incus`) still exists
|
||||
- But no Incus storage pool, network, or profile devices are created
|
||||
- OC's Terraform handles all resource creation cleanly during cluster formation
|
||||
|
||||
**Tested result**: cluster forms cleanly with `apply_defaults: false`. OC's
|
||||
Terraform creates the storage pool (`local`), networks (`incusbr0`, `meshbr0`),
|
||||
and injects the client certificate via `--application-seed-config`. The only
|
||||
Terraform error is "Certificate already in trust store" when the cert was also
|
||||
injected via SEED_DATA -- avoid this by only injecting the cert through the
|
||||
application seed config, not the SEED_DATA.
|
||||
|
||||
### apply_defaults: true (also works, with caveats)
|
||||
|
||||
When using `apply_defaults: true`, each node already has a `local` ZFS storage
|
||||
pool, `incusbr0` network bridge, and the client certificate. OC's Terraform
|
||||
then fails with:
|
||||
|
||||
- "Certificate already in trust store"
|
||||
- "Network is not in pending state"
|
||||
- "Storage pool is not in pending state"
|
||||
|
||||
**The cluster itself forms successfully** despite these Terraform errors. All
|
||||
3 nodes join the cluster correctly. The Terraform errors are non-fatal because
|
||||
the resources already exist. However, the `cluster artifact list` will be empty
|
||||
since Terraform didn't complete.
|
||||
|
||||
**Recommended approaches** for future deployments:
|
||||
1. **Keep apply_defaults** (current tested approach): cluster forms fine, ignore
|
||||
Terraform errors. Skip `--application-seed-config` since the cert is already
|
||||
injected by the seed.
|
||||
2. **Skip apply_defaults**: deploy nodes without `apply_defaults: true` and
|
||||
without cert injection in the seed. Let OC handle everything via Terraform.
|
||||
This is the "clean" approach but hasn't been tested yet.
|
||||
**The cluster itself forms successfully** despite these Terraform errors.
|
||||
However, `cluster artifact list` will be empty since Terraform didn't complete.
|
||||
|
||||
### Verify cluster (tested)
|
||||
|
||||
|
|
@ -433,6 +439,59 @@ The proper lifecycle for maintenance is:
|
|||
4. Wait for IncusOS to boot (~60-90s)
|
||||
5. `incus cluster restore <remote>:<member> --force`
|
||||
|
||||
### IncusOS scrub_schedule fix via API (tested)
|
||||
|
||||
IncusOS has an intermittent bug (~50% of deploys) where `state.txt` is written
|
||||
with an empty `scrub_schedule` field. The `registerJobs()` function calls
|
||||
`gocron.IsValid("")` which fails, crashing the daemon in a loop. The REST API
|
||||
is briefly reachable during each crash cycle (Incus starts before the scheduler).
|
||||
|
||||
**Fix via IncusOS REST API** (through the Incus proxy at `/os/` prefix):
|
||||
|
||||
```bash
|
||||
# Check current storage config
|
||||
curl -sk --cert ~/.config/incus/client.crt \
|
||||
--key ~/.config/incus/client.key \
|
||||
https://<NODE_IP>:8443/os/1.0/system/storage
|
||||
|
||||
# If scrub_schedule is empty, fix it
|
||||
curl -sk --cert ~/.config/incus/client.crt \
|
||||
--key ~/.config/incus/client.key \
|
||||
-X PUT -H "Content-Type: application/json" \
|
||||
-d '{"config":{"scrub_schedule":"0 4 * * 0"}}' \
|
||||
https://<NODE_IP>:8443/os/1.0/system/storage
|
||||
```
|
||||
|
||||
**Key details**:
|
||||
- The `/os/` prefix is required — this proxies to the IncusOS daemon API
|
||||
- PATCH returns 501 (not implemented); use PUT instead
|
||||
- `incus admin os system storage edit` fails because it reads the current
|
||||
(corrupt) config and validates it before allowing edits
|
||||
- `incusos-proxmox` now includes `fix_scrub_schedule()` which proactively
|
||||
fixes this on every deployed node (safe — returns early if already set)
|
||||
- Filed upstream as IncusOS issue #843
|
||||
|
||||
### Node failure and recovery (tested)
|
||||
|
||||
Proxmox hard-stop simulates node crash. Incus cluster detects failure via
|
||||
heartbeat timeout (~30-40s). Node auto-rejoins after Proxmox restart (~60s boot):
|
||||
|
||||
```bash
|
||||
# Simulate crash: Proxmox API stop
|
||||
curl -sk -X POST ".../nodes/pve/qemu/<VMID>/status/stop" ...
|
||||
|
||||
# After ~40s: incus cluster list shows OFFLINE with heartbeat info
|
||||
|
||||
# Restart: Proxmox API start
|
||||
curl -sk -X POST ".../nodes/pve/qemu/<VMID>/status/start" ...
|
||||
|
||||
# After ~60s boot: node auto-rejoins cluster (ONLINE)
|
||||
# Containers on the node auto-start after recovery
|
||||
```
|
||||
|
||||
**OC does not detect node failures**: `provisioning server show` still shows
|
||||
`Status: ready` for an OFFLINE/crashed node. `Last Seen` timestamp is stale.
|
||||
|
||||
**OC does not track Incus cluster member state**: the server list always
|
||||
shows `ready` status even when the Incus cluster reports `EVACUATED`.
|
||||
|
||||
|
|
@ -542,6 +601,7 @@ printf "yes\n" | incus cluster remove oc-node-01:oc-node-03 --force
|
|||
# 2. Destroy broken VM via Proxmox (stop + delete)
|
||||
|
||||
# 3. Redeploy via incusos-proxmox --iso (single-node config)
|
||||
# Use apply_defaults: false for OC-managed nodes
|
||||
./incusos/incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes /tmp/redeploy-node-03.yaml
|
||||
|
||||
# 4. New node auto-registers with OC (same hostname, new IP)
|
||||
|
|
@ -550,18 +610,10 @@ printf "yes\n" | incus cluster remove oc-node-01:oc-node-03 --force
|
|||
|
||||
# 5. Manual cluster join (since node was force-removed)
|
||||
incus config set oc-node-03: core.https_address <NEW_IP>:8443
|
||||
# Remove apply_defaults resources
|
||||
incus config unset oc-node-03: storage.backups_volume
|
||||
incus config unset oc-node-03: storage.images_volume
|
||||
incus storage volume delete oc-node-03:local backups
|
||||
incus storage volume delete oc-node-03:local images
|
||||
incus profile device remove oc-node-03:default root
|
||||
incus profile device remove oc-node-03:default eth0
|
||||
incus storage delete oc-node-03:local
|
||||
incus network delete oc-node-03:incusbr0
|
||||
# With apply_defaults: false, no cleanup needed (no pool/network to delete)
|
||||
# Generate token and join
|
||||
incus cluster add oc-node-01:oc-node-03
|
||||
printf '\n\nyes\n\nlocal/incus\nlocal/incus\n' | incus cluster join oc-node-01: oc-node-03:
|
||||
printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join oc-node-01: oc-node-03:
|
||||
# Note: extra \n for meshbr0 tunnel.mesh.interface prompt (added by OC clustering)
|
||||
|
||||
# 6. Fix remote after cert change
|
||||
|
|
@ -569,6 +621,21 @@ incus remote remove oc-node-03
|
|||
incus remote add oc-node-03 https://<NEW_IP>:8443 --accept-certificate
|
||||
```
|
||||
|
||||
### Node failure simulation (tested)
|
||||
|
||||
Proxmox hard-stop simulates node crash. Incus detects via heartbeat (~40s):
|
||||
|
||||
```bash
|
||||
# Simulate crash
|
||||
curl -sk -X POST ".../nodes/pve/qemu/<VMID>/status/stop" ...
|
||||
# incus cluster list shows OFFLINE after ~40s
|
||||
|
||||
# Restart
|
||||
curl -sk -X POST ".../nodes/pve/qemu/<VMID>/status/start" ...
|
||||
# Node auto-rejoins after ~60s boot. Containers auto-start.
|
||||
# OC still shows "ready" (does not detect node failures)
|
||||
```
|
||||
|
||||
### Factory reset (not tested)
|
||||
|
||||
```bash
|
||||
|
|
@ -650,10 +717,20 @@ operations-center provisioning cluster factory-reset oc-cluster <token-uuid>
|
|||
| `inventory instance list` | List all instances |
|
||||
| `inventory instance show <UUID>` | Instance details (by UUID, not name) |
|
||||
| `inventory storage-pool list` | List storage pools |
|
||||
| `inventory storage-volume list` | List storage volumes |
|
||||
| `inventory storage-bucket list` | List storage buckets |
|
||||
| `inventory network list` | List networks |
|
||||
| `inventory network-acl list` | List network ACLs |
|
||||
| `inventory network-address-set list` | List network address sets |
|
||||
| `inventory network-forward list` | List network forwards |
|
||||
| `inventory network-integration list` | List network integrations |
|
||||
| `inventory network-load-balancer list` | List network load balancers |
|
||||
| `inventory network-peer list` | List network peers |
|
||||
| `inventory network-zone list` | List network zones |
|
||||
| `inventory profile list` | List profiles |
|
||||
| `inventory project list` | List projects |
|
||||
| `inventory image list` | List images |
|
||||
| `inventory query --cluster <name>` | Cross-resource tree view (v0.3.0+) |
|
||||
|
||||
### Updates
|
||||
|
||||
|
|
@ -683,6 +760,8 @@ operations-center provisioning cluster factory-reset oc-cluster <token-uuid>
|
|||
| `admin os service list` | List services (iscsi, lvm, nvme, ovn, tailscale, ...) |
|
||||
| `admin os service show <name>` | Service config |
|
||||
| `admin os debug log` | Debug logs |
|
||||
| `admin os debug processes` | System process list |
|
||||
| `admin os debug secureboot` | Secure Boot debug info |
|
||||
|
||||
All commands prefixed with `operations-center`.
|
||||
|
||||
|
|
@ -713,6 +792,19 @@ curl -sk --cert ~/.config/incus/client.crt \
|
|||
| GET | `/1.0/provisioning/clusters` | List clusters |
|
||||
| POST | `/1.0/provisioning/clusters` | Form cluster |
|
||||
| GET | `/1.0/inventory/instances` | List all instances |
|
||||
| GET | `/1.0/inventory/instance/<UUID>` | Instance details |
|
||||
| GET | `/1.0/provisioning/servers/<name>` | Server details + hardware inventory |
|
||||
|
||||
### IncusOS REST API (via Incus proxy)
|
||||
|
||||
IncusOS nodes expose their own REST API through the Incus proxy at the `/os/`
|
||||
prefix on port 8443:
|
||||
|
||||
| Method | Endpoint | Description |
|
||||
|--------|----------|-------------|
|
||||
| GET | `/os/1.0/system/storage` | Storage config (scrub_schedule) |
|
||||
| PUT | `/os/1.0/system/storage` | Update storage config |
|
||||
| GET | `/os/1.0` | IncusOS system info |
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -771,7 +863,7 @@ ip neigh show | grep -i "$MAC" | awk '{print $1}'
|
|||
|
||||
## Known Limitations (Tested Findings)
|
||||
|
||||
### Confirmed limitations (v0.2.2)
|
||||
### Confirmed limitations (v0.3.0)
|
||||
|
||||
- **No brownfield adoption**: nodes must boot from OC-provisioned ISO
|
||||
- **No real-time inventory sync**: requires explicit `cluster resync`
|
||||
|
|
@ -869,20 +961,24 @@ ip neigh show | grep -i "$MAC" | awk '{print $1}'
|
|||
|
||||
## Open Research Items
|
||||
|
||||
These are observed behaviors whose root cause is not yet confirmed:
|
||||
|
||||
- **OC reboot failure mechanism** (resolved): guest reboot is safe on
|
||||
standalone IncusOS nodes (tested: simultaneous 3-node reboot, all recover
|
||||
in ~50s). The failure is OC-specific: the OC agent runs on every boot and
|
||||
fails with configuration errors (invalid crontab), re-registration
|
||||
conflicts (constraint violation), or cluster quorum loss (peers in boot
|
||||
loops). The root cause is NOT dqlite corruption or TPM issues.
|
||||
in ~50s). The failure is OC-specific: the OC agent pushes config via the
|
||||
IncusOS REST API that gets persisted to `state.txt`. On reboot, invalid
|
||||
values (e.g., cron expression where Go duration is expected) crash the
|
||||
daemon. Root cause is NOT dqlite corruption or TPM issues.
|
||||
|
||||
- **apply_defaults conflict with OC Terraform**: when nodes have
|
||||
`apply_defaults: true`, OC's post-clustering Terraform step fails with
|
||||
"already in trust store" / "not in pending state" errors. The cluster forms
|
||||
successfully despite these errors. Unclear whether skipping `apply_defaults`
|
||||
would allow clean Terraform completion (untested).
|
||||
- **apply_defaults conflict with OC Terraform** (resolved): `apply_defaults: false`
|
||||
is now the tested and recommended approach. OC's Terraform handles storage pool,
|
||||
network, and cert creation cleanly when nodes start without defaults.
|
||||
`apply_defaults: true` also works but produces non-fatal Terraform errors.
|
||||
|
||||
- **IncusOS scrub_schedule bug** (workaround found, upstream filed as #843):
|
||||
~50% of deploys hit an empty `scrub_schedule` in `state.txt`. Root cause is
|
||||
likely a race condition between REST API startup and `registerJobs()`, or a
|
||||
zero-value encoding bug in `encodeHelper()`. Workaround: `PUT /os/1.0/system/storage`
|
||||
with `{"config":{"scrub_schedule":"0 4 * * 0"}}`. `incusos-proxmox` now
|
||||
auto-heals this proactively on every deploy.
|
||||
|
||||
- **Stale server entries after out-of-band changes**: when cluster members are
|
||||
removed via `incus cluster remove` (not through OC), stale entries persist in
|
||||
|
|
@ -893,6 +989,10 @@ These are observed behaviors whose root cause is not yet confirmed:
|
|||
to pick up changes made via the `incus` CLI. No webhook or event-driven sync.
|
||||
This applies to instance creation, migration, evacuation, and deletion.
|
||||
|
||||
- **Node failure detection**: OC does not detect node failures. `provisioning
|
||||
server show` continues to report `Status: ready` for OFFLINE/crashed nodes.
|
||||
Incus cluster detects failures via heartbeat (~40s), but OC has no equivalent.
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
|
|
|||
Loading…
Reference in New Issue