incus-contrib/notes/aether-inventory-post-redeploy.md

Aether Inventory After Redeploy (Without DB Backup)

Date: 2026-03-04 Target: Hetzner lab, Aether at 10.10.0.120 Version: 6.4.440 (up from 6.4.202 at time of redeploy) Activation code: 443F4EEC6E088017 (new — DB reinstall generates a new one)

Summary

Aether was redeployed from a golden image without restoring the previous database. This inventory documents what survived (auto-discovered from the live cluster) versus what was lost (stored only in the DB).

What survived (auto-discovered from cluster)

These are things Aether reconstructs by connecting to the Incus cluster and querying the API. They require no database state.

Feature	Status	Detail
Cluster connection	Present	hz-cluster (ID 53, was 52 before redeploy)
Cluster members	Present	3/3 nodes online, fully operational
Instance inventory	Present	12 instances, all Running, correct IPs and locations
Instance tags	Present	HAProxy tags (ha-group, managed, role) visible
Storage pools	Present	local (zfs), 6% used (7GB/112.4GB)
Networking/OVN	Present	Visible via Manage INCUS Clusters > Networking
Cluster health	Present	Health dashboard shows CPU/mem/load per node

Key insight: Aether's core value — seeing your cluster and instances — works immediately after a fresh deploy + cluster add. The Incus API is the source of truth for all infrastructure state; Aether caches it but can rebuild from scratch.

What was lost (DB-only state)

These are things Aether stores in its PostgreSQL database that have no external source of truth. They must be manually recreated.

Feature	Status	Detail
Blueprints	Empty	"No blueprints created yet" — all designs gone
Deployed blueprints	Empty	No deployment history — tracking of which blueprints were deployed where is lost
Operations Center connections	Empty	No OC configured — previous OC link gone
Global firewall rules	Empty	0 rules (API confirms: [])
Cluster firewall rules	Empty	0 rules (API confirms: [])
RBAC users/roles	Default only	Only the built-in admin account exists
License	Gone	New activation code generated — previous license key invalid
Ledger / audit log	Empty	No historical log entries
Sync logs	Empty	No sync history
AWX job history	Empty	"No AWX job history yet" — past job runs not tracked
HAProxy image push state	Not pushed	Image v2.64 exists locally but shows "No Image" for hz-cluster — needs re-push
Auto-backup setting	Off	auto_backup_enabled: False (default)

What was re-added manually (post-redeploy)

These items exist in the current database because they were manually reconfigured after the redeploy.

Feature	Status	Detail
AWX endpoint	Configured	"lab-awx" at http://10.10.0.122:30080, created 2026-03-03 11:17:36
AWX cluster config	Configured	hz-cluster linked to lab-awx, post-deploy template 9, decommission template 10, 600s timeout
AWX health	Healthy	Version 24.6.1, reachable

Observations

Cluster ID changed (52 → 53)

The previous Aether instance had the cluster registered as ID 52 (visible in haproxy.yaml and awx.yaml configs). After redeploy, the new database assigned it ID 53. This is a synthetic auto-increment ID internal to Aether's DB. The cluster itself (certificate, nodes, instances) is unchanged — only Aether's internal reference number differs.

Impact: Scripts that hardcode cluster_id: 52 (haproxy.yaml, awx.yaml) would need updating if they interact with Aether's API using this ID. The deploy scripts use the cluster_id for Aether API calls (HAProxy image push, AWX cluster config). If these scripts are re-run against this Aether instance, the ID mismatch would cause failures.

HAProxy image survived, push state didn't

The HAProxy base image v2.64 (287 MB, built 2026-02-21) is stored on Aether's local filesystem, not in the DB. It survived the redeploy. But the DB record of which clusters it was pushed to was lost. The UI shows "No Image" for hz-cluster, meaning the image needs to be re-pushed before HAProxy LB management works through Aether.

The HAProxy containers themselves (ffsdn-haproxy-52-01/02) are running fine in the cluster — they don't depend on Aether for runtime operation. Aether only manages their configuration and deployment lifecycle.

Settings are all defaults

All settings reverted to defaults. Notable:

• log_level: Debug (default, verbose — consider changing to Info)
• refresh_interval_minutes: 5
• auto_backup_enabled: False — should enable this time
• manage_instances: True
• manage_vcenters: False
• migration_manager: False

What this tells us about Aether's architecture

1. Cluster state is ephemeral in Aether — it's a cache of the Incus API. Losing the DB doesn't lose infrastructure visibility. Re-add the cluster and everything reappears within one sync cycle (5 min default).

2. Operational config is DB-only — blueprints, ACL rules, RBAC, AWX endpoints, OC connections, and deployment history live exclusively in PostgreSQL. No export/import mechanism was observed.

3. The license is tied to the DB — reinstalling generates a new activation code, invalidating any existing license key. This is explicitly noted on the licensing page.

4. HAProxy images are filesystem-based — they survive DB loss but lose their cluster push state. The containers in the cluster are independent of Aether once deployed.

5. AWX integration is lightweight — just an endpoint URL + template IDs. Quick to re-add manually (as was done). Job history is lost but AWX itself retains its own job history.

Recovery priority if this happens again

1. Enable auto_backup_enabled immediately after setup
2. Periodically copy /opt/ffsdn/backups/ off the Aether VM
3. After restore: re-add cluster, re-add AWX endpoint, re-push HAProxy image
4. Blueprints and ACL rules would need to be recreated from scratch (consider documenting them externally)