incus-contrib/notes/aether-inventory-post-redeploy.md

# Aether Inventory After Redeploy (Without DB Backup)

**Date**: 2026-03-04
**Target**: Hetzner lab, Aether at 10.10.0.120
**Version**: 6.4.440 (up from 6.4.202 at time of redeploy)
**Activation code**: 443F4EEC6E088017 (new — DB reinstall generates a new one)

## Summary

Aether was redeployed from a golden image without restoring the previous
database. This inventory documents what survived (auto-discovered from
the live cluster) versus what was lost (stored only in the DB).

### What survived (auto-discovered from cluster)

These are things Aether reconstructs by connecting to the Incus cluster
and querying the API. They require no database state.

| Feature | Status | Detail |
|---------|--------|--------|
| Cluster connection | **Present** | hz-cluster (ID 53, was 52 before redeploy) |
| Cluster members | **Present** | 3/3 nodes online, fully operational |
| Instance inventory | **Present** | 12 instances, all Running, correct IPs and locations |
| Instance tags | **Present** | HAProxy tags (ha-group, managed, role) visible |
| Storage pools | **Present** | local (zfs), 6% used (7GB/112.4GB) |
| Networking/OVN | **Present** | Visible via Manage INCUS Clusters > Networking |
| Cluster health | **Present** | Health dashboard shows CPU/mem/load per node |

**Key insight**: Aether's core value — seeing your cluster and instances —
works immediately after a fresh deploy + cluster add. The Incus API is the
source of truth for all infrastructure state; Aether caches it but can
rebuild from scratch.

### What was lost (DB-only state)

These are things Aether stores in its PostgreSQL database that have no
external source of truth. They must be manually recreated.

| Feature | Status | Detail |
|---------|--------|--------|
| Blueprints | **Empty** | "No blueprints created yet" — all designs gone |
| Deployed blueprints | **Empty** | No deployment history — tracking of which blueprints were deployed where is lost |
| Operations Center connections | **Empty** | No OC configured — previous OC link gone |
| Global firewall rules | **Empty** | 0 rules (API confirms: `[]`) |
| Cluster firewall rules | **Empty** | 0 rules (API confirms: `[]`) |
| RBAC users/roles | **Default only** | Only the built-in admin account exists |
| License | **Gone** | New activation code generated — previous license key invalid |
| Ledger / audit log | **Empty** | No historical log entries |
| Sync logs | **Empty** | No sync history |
| AWX job history | **Empty** | "No AWX job history yet" — past job runs not tracked |
| HAProxy image push state | **Not pushed** | Image v2.64 exists locally but shows "No Image" for hz-cluster — needs re-push |
| Auto-backup setting | **Off** | `auto_backup_enabled: False` (default) |

### What was re-added manually (post-redeploy)

These items exist in the current database because they were manually
reconfigured after the redeploy.

| Feature | Status | Detail |
|---------|--------|--------|
| AWX endpoint | **Configured** | "lab-awx" at http://10.10.0.122:30080, created 2026-03-03 11:17:36 |
| AWX cluster config | **Configured** | hz-cluster linked to lab-awx, post-deploy template 9, decommission template 10, 600s timeout |
| AWX health | **Healthy** | Version 24.6.1, reachable |

## Observations

### Cluster ID changed (52 → 53)

The previous Aether instance had the cluster registered as ID 52 (visible
in haproxy.yaml and awx.yaml configs). After redeploy, the new database
assigned it ID 53. This is a synthetic auto-increment ID internal to
Aether's DB. The cluster itself (certificate, nodes, instances) is
unchanged — only Aether's internal reference number differs.

**Impact**: Scripts that hardcode `cluster_id: 52` (haproxy.yaml, awx.yaml)
would need updating if they interact with Aether's API using this ID.
The deploy scripts use the cluster_id for Aether API calls (HAProxy image
push, AWX cluster config). If these scripts are re-run against this Aether
instance, the ID mismatch would cause failures.

### HAProxy image survived, push state didn't

The HAProxy base image v2.64 (287 MB, built 2026-02-21) is stored on
Aether's local filesystem, not in the DB. It survived the redeploy. But
the DB record of which clusters it was pushed to was lost. The UI shows
"No Image" for hz-cluster, meaning the image needs to be re-pushed before
HAProxy LB management works through Aether.

The HAProxy containers themselves (ffsdn-haproxy-52-01/02) are running
fine in the cluster — they don't depend on Aether for runtime operation.
Aether only manages their configuration and deployment lifecycle.

### Settings are all defaults

All settings reverted to defaults. Notable:
- `log_level: Debug` (default, verbose — consider changing to Info)
- `refresh_interval_minutes: 5`
- `auto_backup_enabled: False` — **should enable this time**
- `manage_instances: True`
- `manage_vcenters: False`
- `migration_manager: False`

### What this tells us about Aether's architecture

1. **Cluster state is ephemeral in Aether** — it's a cache of the Incus API.
   Losing the DB doesn't lose infrastructure visibility. Re-add the cluster
   and everything reappears within one sync cycle (5 min default).

2. **Operational config is DB-only** — blueprints, ACL rules, RBAC, AWX
   endpoints, OC connections, and deployment history live exclusively in
   PostgreSQL. No export/import mechanism was observed.

3. **The license is tied to the DB** — reinstalling generates a new
   activation code, invalidating any existing license key. This is
   explicitly noted on the licensing page.

4. **HAProxy images are filesystem-based** — they survive DB loss but
   lose their cluster push state. The containers in the cluster are
   independent of Aether once deployed.

5. **AWX integration is lightweight** — just an endpoint URL + template IDs.
   Quick to re-add manually (as was done). Job history is lost but AWX
   itself retains its own job history.

## Recovery priority if this happens again

1. Enable `auto_backup_enabled` immediately after setup
2. Periodically copy `/opt/ffsdn/backups/` off the Aether VM
3. After restore: re-add cluster, re-add AWX endpoint, re-push HAProxy image
4. Blueprints and ACL rules would need to be recreated from scratch
   (consider documenting them externally)