Update Hetzner configs for cluster_id=1 after Aether redeploy

- Switch awx/haproxy/observability configs + post-deploy playbook to the
  fresh Aether cluster (id=1, 10.10.0.111)
- Rename HAProxy instances from ffsdn-haproxy-52-* to ffsdn-haproxy-1-*
- Add bin/lab-status script and Aether post-redeploy inventory notes
- Ignore .claude/scheduled_tasks.lock

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

.claude/rules/aether-api-payloads.md

@@ -1,7 +1,7 @@
 # Aether API Payloads Context
 
 paths:
-  - incus-mitm
+  - bin/helpers/incus-mitm
   - api-interception
 
 ## How Aether Communicates with Incus
@@ -20,11 +20,11 @@ prefix is `6cfd2a1949a7`.
 
 ```bash
 # Live stream with Aether filter
-incusos/helpers/incus-mitm --live --filter aether
+bin/helpers/incus-mitm --live --filter aether
 
 # Capture to file
-incusos/helpers/incus-mitm --capture 120
-incusos/helpers/incus-mitm --analyze /tmp/incus-events-*.json --filter aether
+bin/helpers/incus-mitm --capture 120
+bin/helpers/incus-mitm --analyze /tmp/incus-events-*.json --filter aether
 ```
 
 ### API Call Patterns

.claude/rules/awx-integration.md

@@ -1,8 +1,8 @@
 ---
 paths:
   - "ansible/**"
-  - "incusos/deploy-awx"
-  - "incusos/awx-manifests/*"
+  - "bin/deploy-awx"
+  - "bin/data/awx-manifests/*"
   - "notes/awx-guide.md"
 ---
 

.claude/rules/clustering.md

@@ -1,7 +1,7 @@
 ---
 paths:
-  - "incusos/lab-test"
-  - "incusos/incusos-proxmox"
+  - "bin/lab-test"
+  - "bin/incusos-proxmox"
   - "notes/clustering-guide.md"
   - "notes/production-lab-guide.md"
 ---

.claude/rules/haproxy-lb.md

@@ -1,7 +1,7 @@
 ---
 paths:
-  - "incusos/deploy-haproxy"
-  - "incusos/helpers/aether-browser"
+  - "bin/deploy-haproxy"
+  - "bin/helpers/aether-browser"
   - "notes/haproxy-guide.md"
 ---
 

.claude/rules/hetzner-setup.md

@@ -1,8 +1,8 @@
 # Hetzner Setup Context
 
 paths:
-  - hetzner/*
-  - incusos/targets/hetzner/*
+  - envs/hetzner/*
+  - envs/hetzner/setup/*
 
 ## Network Topology
 
@@ -35,7 +35,7 @@ VMs cannot use public IPs directly.
 
 ### proxmox-setup Script
 
-`hetzner/proxmox-setup` runs from the workstation over SSH.
+`envs/hetzner/setup/proxmox-setup` runs from the workstation over SSH.
 All commands executed via `ssh <host> <command>`.
 Follows same conventions as other scripts: setup_colors, info/warn/error,
 --dry-run, --yes flags.

.claude/rules/incusos-immutability.md

@@ -1,6 +1,6 @@
 ---
 paths:
-  - "incusos/helpers/incusos-health"
+  - "bin/helpers/incusos-health"
   - "notes/incusos-break-fix.md"
 ---
 

.claude/rules/incusos-scripts.md

@@ -1,10 +1,10 @@
 ---
 paths:
-  - "incusos/incusos-iso"
-  - "incusos/incusos-seed"
-  - "incusos/incusos-proxmox"
-  - "incusos/helpers/*"
-  - "incusos/examples/*"
+  - "bin/incusos-iso"
+  - "bin/incusos-seed"
+  - "bin/incusos-proxmox"
+  - "bin/helpers/*"
+  - "bin/examples/*"
 ---
 
 # IncusOS Scripts Context

.claude/rules/lab-infrastructure.md

@@ -1,8 +1,8 @@
 ---
 paths:
-  - "incusos/incusos-proxmox"
-  - "incusos/lab-test"
-  - "incusos/examples/*"
+  - "bin/incusos-proxmox"
+  - "bin/lab-test"
+  - "bin/examples/*"
 ---
 
 # Lab Infrastructure

.claude/rules/observability.md

@@ -1,7 +1,7 @@
 ---
 paths:
-  - "incusos/deploy-observability"
-  - "incusos/observability-dashboards"
+  - "bin/deploy-observability"
+  - "bin/data/observability-dashboards"
   - "notes/observability-guide.md"
 ---
 

.claude/rules/operations-center.md

@@ -1,8 +1,8 @@
 ---
 paths:
   - "notes/operations-center-guide.md"
-  - "incusos/incusos-proxmox"
-  - "incusos/examples/*oc*"
+  - "bin/incusos-proxmox"
+  - "bin/examples/*oc*"
 ---
 
 # Operations Center

.claude/rules/ovn-internals.md

@@ -1,8 +1,8 @@
 # OVN Internals Context
 
 paths:
-  - ovn-inspect
-  - ovn-deep-dive
+  - bin/helpers/ovn-inspect
+  - notes/ovn-deep-dive*
 
 ## OVN Architecture in this Cluster
 
@@ -26,7 +26,7 @@ Incus manages OVN objects using a `net8` prefix convention.
 - NB/SB commands: `incus exec oc-node-01:ovn-central -- ovn-nbctl ...`
 - OVS commands: Need privileged container with `/run/openvswitch` mounted
   (deploy Alpine on net-prod, `apk add openvswitch`)
-- Helper script: `incusos/helpers/ovn-inspect --nb|--sb|--ovs|--full|--trace`
+- Helper script: `bin/helpers/ovn-inspect --nb|--sb|--ovs|--full|--trace`
 
 ### Important Details
 

.claude/rules/proxmox-deployment.md

@@ -1,10 +1,10 @@
 ---
 paths:
-  - "incusos/incusos-proxmox"
-  - "incusos/observe-deploy"
-  - "incusos/helpers/*"
-  - "incusos/examples/*"
-  - "incusos/TESTING.md"
+  - "bin/incusos-proxmox"
+  - "bin/diagnostic/observe-deploy"
+  - "bin/helpers/*"
+  - "bin/examples/*"
+  - "bin/TESTING.md"
 ---
 
 # Proxmox VE Deployment

.claude/skills/proxmox-api.md

@@ -16,7 +16,7 @@ Make an authenticated Proxmox API call with proper token handling.
 
 1. Run the helper script:
    ```bash
-   /home/maarten/dev/incus-contrib/incusos/helpers/proxmox-api {{args}}
+   /home/maarten/dev/incus-contrib/bin/helpers/proxmox-api {{args}}
    ```
 2. Parse the JSON response
 3. Present the results in a readable format

.claude/skills/screenshot.md

@@ -16,7 +16,7 @@ Take a console screenshot of a Proxmox VM and display it.
 
 1. Run the helper script to capture the screenshot:
    ```bash
-   /home/maarten/dev/incus-contrib/incusos/helpers/proxmox-screenshot {{vmid}}
+   /home/maarten/dev/incus-contrib/bin/helpers/proxmox-screenshot {{vmid}}
    ```
 2. Read the output path from stdout
 3. Use the Read tool to view the PNG image

.gitignore

@@ -26,13 +26,11 @@ Thumbs.db
 # Go binaries (flasher-tool)
 flasher-tool
 
-# Proxmox connection config (contains credentials)
-proxmox.yaml
+# Per-environment secrets and connection configs
+envs/*/env
+envs/*/proxmox.yaml
 
-# Per-target env files
-env-*
-
-# Environment secrets
+# Legacy: root env file (backward compat)
 env
 
 # Observational deploy output (screenshots, logs)
@@ -43,3 +41,6 @@ sources/
 
 # Playwright MCP logs
 .playwright-mcp/
+
+# Claude Code local state
+.claude/scheduled_tasks.lock

CLAUDE.md

@@ -20,16 +20,16 @@ incus-contrib/
 │   └── playbooks/
 │       ├── post-deploy.yml    # Runs after Aether creates an instance
 │       └── decommission.yml   # Runs before Aether deletes an instance
-├── incusos/               # IncusOS installation tooling
-│   ├── README.md
+├── bin/                   # ALL reusable scripts
 │   ├── incusos-iso        # ISO/IMG builder (wraps flasher-tool)
 │   ├── incusos-seed       # Seed archive generator (Linux + macOS)
 │   ├── incusos-proxmox    # Declarative Proxmox VM deployment + lab lifecycle
+│   ├── lab-test           # Guided lab validation (12 test phases)
 │   ├── deploy-awx         # AWX deployment + management on Incus cluster
 │   ├── deploy-haproxy     # HAProxy LB deployment + management via Aether
 │   ├── deploy-observability # Prometheus + Grafana + Loki observability stack
-│   ├── awx-manifests/     # K8s manifests for AWX Operator + instance
-│   ├── observability-dashboards/  # Grafana dashboard JSON exports
+│   ├── manage-dashboards  # Grafana dashboard management
+│   ├── pve-api            # Python Proxmox API helper
 │   ├── helpers/
 │   │   ├── proxmox-screenshot  # VMID -> PNG console screenshot
 │   │   ├── proxmox-api         # Authenticated API calls (handles ! in token)
@@ -37,18 +37,41 @@ incus-contrib/
 │   │   ├── ovn-inspect         # OVN/OVS topology inspection (--nb/--sb/--ovs/--trace)
 │   │   ├── incus-mitm         # API traffic capture and analysis (--capture/--live/--analyze)
 │   │   └── incusos-health    # IncusOS health inspection (--status/--tpm/--partitions/--all)
-│   ├── lab-test           # Guided lab validation (12 test phases)
-│   ├── observe-deploy     # Single-VM deploy with console screenshots
-│   ├── targets/           # Per-host configs (beelink/, hetzner/)
+│   ├── data/              # Static data consumed by scripts
+│   │   ├── awx-manifests/     # K8s manifests for AWX Operator + instance
+│   │   └── observability-dashboards/  # Grafana dashboard JSON exports
+│   ├── diagnostic/        # Investigation/one-off tools
+│   │   ├── observe-deploy     # Single-VM deploy with console screenshots
+│   │   ├── investigate-boot   # Boot investigation
+│   │   └── test-boot-reliability  # Parallel boot testing
+│   ├── examples/          # Example seed + Proxmox YAML files
+│   └── TESTING.md
+├── envs/                  # Per-environment config + secrets
+│   ├── README.md          # How environments work
+│   ├── beelink/           # Beelink mini PC target
+│   │   ├── env                # Secrets (gitignored)
+│   │   ├── proxmox.yaml       # Proxmox connection (gitignored)
+│   │   ├── proxmox.yaml.example
+│   │   ├── lab-cluster.yaml
+│   │   ├── awx.yaml
+│   │   ├── haproxy.yaml
+│   │   └── observability.yaml
+│   └── hetzner/           # Hetzner dedicated server target
+│       ├── env                # Secrets (gitignored)
 │       ├── proxmox.yaml       # Proxmox connection (gitignored)
-│   ├── TESTING.md
-│   └── examples/          # Example seed + Proxmox YAML files
-├── hetzner/               # Hetzner dedicated server setup
-│   ├── README.md              # Overview, prerequisites, quickstart
-│   ├── hetzner-setup.md       # Step-by-step Proxmox setup guide
-│   └── proxmox-setup          # Interactive setup helper (runs over SSH)
-├── env                    # PROXMOX_TOKEN_SECRET, PROXMOX_ROOT_PASSWORD, AETHER_ADMIN_PASSWORD (gitignored)
-└── notes/                 # Reference guides (clustering, networking, OC, AWX, etc.)
+│       ├── proxmox.yaml.example
+│       ├── lab-cluster.yaml
+│       ├── lab-production.yaml
+│       ├── awx.yaml
+│       ├── haproxy.yaml
+│       ├── observability.yaml
+│       └── setup/             # Host provisioning
+│           ├── README.md
+│           ├── hetzner-setup.md
+│           ├── hetzner-lab-guide.md
+│           └── proxmox-setup
+├── notes/                 # Reference guides (clustering, networking, OC, AWX, etc.)
+└── sources/               # External software (gitignored)
 ```
 
 ## Capabilities you have
@@ -56,7 +79,7 @@ incus-contrib/
 ### Proxmox VM screenshots
 Take console screenshots during deploys to see boot progress or errors:
 ```
-incusos/helpers/proxmox-screenshot VMID [/tmp/output.png]
+bin/helpers/proxmox-screenshot VMID [/tmp/output.png]
 ```
 Use the Read tool on the PNG to view it. **USE THIS PROACTIVELY** during
 deploy scenarios to monitor boot progress, diagnose hangs, and verify installs.
@@ -64,9 +87,9 @@ deploy scenarios to monitor boot progress, diagnose hangs, and verify installs.
 ### Proxmox API calls
 Make authenticated API calls (handles the `!` in `automation@pve!deploy`):
 ```
-incusos/helpers/proxmox-api GET /nodes/pve/qemu/900/status/current --json
-incusos/helpers/proxmox-api GET /nodes/pve/status --json
-incusos/helpers/proxmox-api POST /nodes/pve/qemu/900/status/stop
+bin/helpers/proxmox-api GET /nodes/pve/qemu/900/status/current --json
+bin/helpers/proxmox-api GET /nodes/pve/status --json
+bin/helpers/proxmox-api POST /nodes/pve/qemu/900/status/stop
 ```
 
 ### Aether API
@@ -75,7 +98,7 @@ Aether has a documented REST API at `https://192.168.102.160:8443/api/`.
 - **Swagger UI**: `https://192.168.102.160:8443/api/docs`
 - **OpenAPI spec**: `https://192.168.102.160:8443/api/swagger.yaml`
 - **Auth**: JWT via `POST /api/auth/token` with `{"username":"...","password":"..."}`
-  Credentials in `env` file (`AETHER_ADMIN_PASSWORD`). Token valid 24h.
+  Credentials in `envs/*/env` file (`AETHER_ADMIN_PASSWORD`). Token valid 24h.
 
 ```bash
 # Get JWT
@@ -117,11 +140,11 @@ that curl cannot handle reliably. **USE PLAYWRIGHT** for these interactions.
 `browser_click`, `browser_fill`, `browser_screenshot` etc. as tools when
 the Playwright MCP server is running. Prefer MCP tools when available.
 
-**Helper script**: `incusos/helpers/aether-browser` provides standalone
+**Helper script**: `bin/helpers/aether-browser` provides standalone
 Playwright automation when MCP tools aren't loaded:
 ```bash
-source env  # loads AETHER_ADMIN_PASSWORD
-NODE_PATH=~/node_modules node incusos/helpers/aether-browser <action> [args]
+source envs/beelink/env  # loads AETHER_ADMIN_PASSWORD
+NODE_PATH=~/node_modules node bin/helpers/aether-browser <action> [args]
 ```
 
 Actions:
@@ -177,6 +200,7 @@ still UI-only. The API endpoint may have been added since the guide was written.
 - **Flags**: both short (`-d`) and long (`--defaults`)
 - **Defaults**: useful behavior with zero flags
 - **Dry run**: all scripts support `--dry-run`
+- **Config**: deploy scripts use `--config FILE` for environment-specific settings
 - **Cert detection**: files on disk first (`~/.config/incus/client.crt`), CLI second
 - **Errors**: include actionable remediation steps
 - **No hardcoded package managers**: say "install the Incus client" with a link
@@ -201,7 +225,7 @@ which files are being edited:
 | `incusos-immutability.md` | incusos-health, incusos-break-fix |
 | `proxmox-ssh-rules.md` | **Always loaded** (no paths filter) |
 | `lab-infrastructure.md` | incusos-proxmox, lab-test, examples |
-| `hetzner-setup.md` | hetzner/, incusos/targets/hetzner/ |
+| `hetzner-setup.md` | envs/hetzner/, setup scripts |
 
 For deep reference, see the guides in `notes/`.
 

README.md

@@ -1,4 +1,4 @@
-# incu-contrib
+# incus-contrib
 
 Peripheral tools, snippets, and scripts for working with [Incus](https://linuxcontainers.org/incus/),
 [IncusOS](https://linuxcontainers.org/incus-os/), and the broader ecosystem.
@@ -7,9 +7,10 @@ Primarily aimed at home lab environments but useful in production as well.
 
 ## Contents
 
-### [`incusos/`](incusos/)
+### [`bin/`](bin/)
 
-Scripts for building IncusOS installation media and seed configurations:
+Reusable scripts for building IncusOS installation media, deploying labs,
+and managing services:
 
 - **`incusos-iso`** -- Build customized IncusOS ISO/IMG images using the official flasher-tool.
   Supports both architectures (x86_64, aarch64), multiple applications (Incus, Operations Center),
@@ -23,15 +24,29 @@ Scripts for building IncusOS installation media and seed configurations:
   YAML, and the script handles seed generation, ISO upload, VM creation with correct
   settings (UEFI, TPM 2.0, VirtIO-SCSI), and automated installation.
 
-- **`targets/`** -- Per-host Proxmox connection configs and lab definitions. Each
-  subdirectory (beelink, hetzner) has a `proxmox.yaml.example` and lab YAML files
-  sized for that host. See [`targets/README.md`](incusos/targets/README.md).
-
 - **`deploy-awx`** -- Deploy and manage AWX (Ansible automation) on the Incus
   cluster. Handles VM creation, K3s install, AWX Operator deployment, project/template
-  configuration, and Aether integration.
+  configuration, and Aether integration. Requires `--config` for environment settings.
 
-See [`incusos/README.md`](incusos/README.md) for full documentation.
+- **`deploy-haproxy`** -- Deploy HA HAProxy load balancing via Aether. Requires
+  `--config` for environment settings.
+
+- **`deploy-observability`** -- Deploy Prometheus + Grafana + Loki monitoring stack.
+  Requires `--config` for environment settings.
+
+See [`bin/README.md`](bin/README.md) for full documentation.
+
+### [`envs/`](envs/)
+
+Per-environment configuration, secrets, and deployment settings:
+
+- **`beelink/`** -- Beelink mini PC target (LAN, vmbr0, VLAN 69, local-lvm)
+- **`hetzner/`** -- Hetzner dedicated server target (WireGuard, vmbr1, local-zfs)
+  - **`setup/`** -- Host provisioning guide and interactive setup script
+
+Each environment has: `env` (secrets), `proxmox.yaml` (connection),
+lab YAML files (VM definitions), and service configs (`awx.yaml`, `haproxy.yaml`,
+`observability.yaml`). See [`envs/README.md`](envs/README.md).
 
 ### [`ansible/`](ansible/)
 
@@ -43,18 +58,6 @@ API (not SSH) for all instance operations — works regardless of network topolo
 - **`playbooks/decommission.yml`** -- Runs before Aether deletes an instance.
   Graceful service shutdown and decommission logging (best-effort).
 
-### [`hetzner/`](hetzner/)
-
-Setup guide and automation for deploying IncusOS labs on a Hetzner dedicated server:
-
-- **`hetzner-setup.md`** -- Step-by-step guide: Proxmox install, private networking,
-  WireGuard tunnel, firewall lockdown, and API token creation.
-- **`proxmox-setup`** -- Interactive helper script that automates the guide steps
-  over SSH. Configures repos, bridge, storage, WireGuard, firewall, and API tokens.
-
-See also [`incusos/targets/`](incusos/targets/) for per-host connection configs
-and lab definitions.
-
 ### [`notes/`](notes/)
 
 Guides and reference material (roughly in learning order):
@@ -89,19 +92,27 @@ Guides and reference material (roughly in learning order):
 go install github.com/lxc/incus-os/incus-osd/cmd/flasher-tool@latest
 
 # Build a default IncusOS ISO with your client cert injected
-./incusos/incusos-iso
+bin/incusos-iso
 
 # Build for a specific architecture and application
-./incusos/incusos-iso --arch aarch64 --app operations-center --defaults
+bin/incusos-iso --arch aarch64 --app operations-center --defaults
 
 # Generate all common ISO variants at once
-./incusos/incusos-iso --batch
+bin/incusos-iso --batch
 
 # Generate a standalone seed archive
-./incusos/incusos-seed --app incus --defaults --output my-seed.tar
+bin/incusos-seed --app incus --defaults --output my-seed.tar
 
-# Generate a SEED_DATA FAT image for external boot media
-./incusos/incusos-seed --format fat --app incus --defaults --output seed.img
+# Deploy a cluster on Hetzner
+source envs/hetzner/env
+bin/incusos-proxmox --proxmox envs/hetzner/proxmox.yaml envs/hetzner/lab-production.yaml
+
+# Deploy AWX on Hetzner
+bin/deploy-awx --config envs/hetzner/awx.yaml --deploy
+
+# Deploy HAProxy on beelink
+source envs/beelink/env
+bin/deploy-haproxy --config envs/beelink/haproxy.yaml --deploy
 ```
 
 ## Requirements

ansible/playbooks/post-deploy.yml

@@ -29,7 +29,7 @@
 
   vars:
     # Incus cluster API (any cluster member works)
-    incus_api: "https://192.168.102.140:8443"
+    incus_api: "https://10.10.0.111:8443"
     # AWX runner mounts project at /runner/project/ during job execution
     incus_cert: "/runner/project/incus-client.crt"
     incus_key: "/runner/project/incus-client.key"

bin/README.md

@@ -376,7 +376,7 @@ directory and current directory. Override with `--proxmox FILE`.
 Copy the template to get started:
 
 ```bash
-cp incusos/examples/proxmox.yaml.example incusos/proxmox.yaml
+cp bin/examples/proxmox.yaml.example envs/beelink/proxmox.yaml
 # Edit with your Proxmox host, credentials, and storage settings
 ```
 
@@ -557,25 +557,25 @@ default — prints what it will do, confirms, executes, and reports pass/fail.
 
 ```bash
 # Run all phases (deploy, single-node tests, cluster, workloads, migration)
-./incusos/lab-test examples/lab-cluster.yaml
+bin/lab-test examples/lab-cluster.yaml
 
 # Deploy and verify only
-./incusos/lab-test --phase deploy examples/lab-cluster.yaml
+bin/lab-test --phase deploy examples/lab-cluster.yaml
 
 # Single-node workload tests
-./incusos/lab-test --phase single examples/lab-cluster.yaml
+bin/lab-test --phase single examples/lab-cluster.yaml
 
 # Form cluster from deployed nodes
-./incusos/lab-test --phase cluster examples/lab-cluster.yaml
+bin/lab-test --phase cluster examples/lab-cluster.yaml
 
 # Test migration and evacuation
-./incusos/lab-test --phase migrate examples/lab-cluster.yaml
+bin/lab-test --phase migrate examples/lab-cluster.yaml
 
 # Full run without prompts
-./incusos/lab-test --yes examples/lab-cluster.yaml
+bin/lab-test --yes examples/lab-cluster.yaml
 
 # Clean up test instances (keeps VMs)
-./incusos/lab-test --cleanup examples/lab-cluster.yaml
+bin/lab-test --cleanup examples/lab-cluster.yaml
 ```
 
 ### Phases

bin/TESTING.md

@@ -140,17 +140,16 @@ key auth first (`ssh-copy-id root@<host>`).
 ### Step 2: Set Up Proxmox Connection
 
 ```bash
-cp incusos/examples/proxmox.yaml.example incusos/proxmox.yaml
+cp bin/examples/proxmox.yaml.example envs/beelink/proxmox.yaml
 ```
 
-Edit `incusos/proxmox.yaml` with your Proxmox host settings. This is
-auto-discovered by the scripts and avoids repeating credentials in each
-lab config.
+Edit `envs/beelink/proxmox.yaml` with your Proxmox host settings, or
+use `--proxmox envs/beelink/proxmox.yaml` to point at it explicitly.
 
 ### Step 3: Create a Test Config
 
 ```bash
-cp incusos/examples/proxmox-minimal.yaml my-test.yaml
+cp bin/examples/proxmox-minimal.yaml my-test.yaml
 ```
 
 Edit `my-test.yaml` -- change only the host IP (or rely on proxmox.yaml):
@@ -176,7 +175,7 @@ Using VMID 900+ avoids clashing with existing VMs.
 ### Step 4: Dry Run
 
 ```bash
-./incusos/incusos-proxmox --dry-run my-test.yaml
+bin/incusos-proxmox --dry-run my-test.yaml
 ```
 
 **Verify in the output:**
@@ -193,7 +192,7 @@ Using VMID 900+ avoids clashing with existing VMs.
 ### Step 5: Deploy (Single VM)
 
 ```bash
-./incusos/incusos-proxmox my-test.yaml
+bin/incusos-proxmox my-test.yaml
 ```
 
 **Watch the phases:**
@@ -219,7 +218,7 @@ Using VMID 900+ avoids clashing with existing VMs.
 After the deploy completes, verify with the built-in status command:
 
 ```bash
-./incusos/incusos-proxmox --status my-test.yaml
+bin/incusos-proxmox --status my-test.yaml
 ```
 
 **Verify in the output:**
@@ -256,7 +255,7 @@ check the Proxmox console — stale ARP entries can be misleading.
 Re-run the deploy to verify the reconcile menu:
 
 ```bash
-./incusos/incusos-proxmox my-test.yaml
+bin/incusos-proxmox my-test.yaml
 ```
 
 **Verify:** The script detects the existing VM and shows the interactive menu.
@@ -265,7 +264,7 @@ Choose option 1 to run status checks without modifying anything.
 Also test with `--yes` to verify it defaults to safe behavior:
 
 ```bash
-./incusos/incusos-proxmox --yes my-test.yaml
+bin/incusos-proxmox --yes my-test.yaml
 ```
 
 **Verify:** Runs post-deployment checks, does NOT destroy.
@@ -273,7 +272,7 @@ Also test with `--yes` to verify it defaults to safe behavior:
 ### Step 9: Clean Up the Test VM
 
 ```bash
-./incusos/incusos-proxmox --cleanup my-test.yaml
+bin/incusos-proxmox --cleanup my-test.yaml
 ```
 
 - Confirms before destroying
@@ -346,9 +345,9 @@ proxmox:
 Deploy and verify:
 
 ```bash
-./incusos/incusos-proxmox --dry-run my-test.yaml   # Should show pool in plan
-./incusos/incusos-proxmox my-test.yaml              # VM created in pool
-./incusos/incusos-proxmox --status my-test.yaml     # Status scoped to pool
+bin/incusos-proxmox --dry-run my-test.yaml   # Should show pool in plan
+bin/incusos-proxmox my-test.yaml              # VM created in pool
+bin/incusos-proxmox --status my-test.yaml     # Status scoped to pool
 ```
 
 ### Step 12: (Optional) Full Lab Deployment
@@ -356,10 +355,10 @@ Deploy and verify:
 Once single-VM tests pass, deploy the full 4-VM lab:
 
 ```bash
-cp incusos/examples/proxmox-lab.yaml my-lab.yaml
+cp bin/examples/proxmox-lab.yaml my-lab.yaml
 # Edit: change host IP, adjust start_vmid if needed
-./incusos/incusos-proxmox --dry-run my-lab.yaml
-./incusos/incusos-proxmox my-lab.yaml
+bin/incusos-proxmox --dry-run my-lab.yaml
+bin/incusos-proxmox my-lab.yaml
 ```
 
 This deploys 1 Operations Center + 3 Incus nodes. After deployment:
@@ -379,44 +378,44 @@ VMs: single-node workloads, cluster formation, migration, and evacuation.
 
 ```bash
 # 1. Check environment
-./incusos/incusos-proxmox --doctor
+bin/incusos-proxmox --doctor
 
 # 2. Deploy 3 nodes
-./incusos/incusos-proxmox --yes examples/lab-cluster.yaml
+bin/incusos-proxmox --yes examples/lab-cluster.yaml
 
 # 3. Add incus remotes for each node (use IPs from --status)
-./incusos/incusos-proxmox --status examples/lab-cluster.yaml
+bin/incusos-proxmox --status examples/lab-cluster.yaml
 incus remote add incus-lab-01 <IP1> --accept-certificate
 incus remote add incus-lab-02 <IP2> --accept-certificate
 incus remote add incus-lab-03 <IP3> --accept-certificate
 
 # 4. Run full lab validation
-./incusos/lab-test --yes examples/lab-cluster.yaml
+bin/lab-test --yes examples/lab-cluster.yaml
 ```
 
 ### Phase-by-Phase Testing
 
 ```bash
 # Single-node workloads only (container + VM launch/exec)
-./incusos/lab-test --phase single --skip-deploy examples/lab-cluster.yaml
+bin/lab-test --phase single --skip-deploy examples/lab-cluster.yaml
 
 # Cluster formation only
-./incusos/lab-test --phase cluster --skip-deploy examples/lab-cluster.yaml
+bin/lab-test --phase cluster --skip-deploy examples/lab-cluster.yaml
 
 # Cluster workloads (scheduler placement + targeted placement)
-./incusos/lab-test --phase workload --skip-deploy examples/lab-cluster.yaml
+bin/lab-test --phase workload --skip-deploy examples/lab-cluster.yaml
 
 # Migration and evacuation tests
-./incusos/lab-test --phase migrate --skip-deploy examples/lab-cluster.yaml
+bin/lab-test --phase migrate --skip-deploy examples/lab-cluster.yaml
 
 # Extended phases (require a working cluster):
-./incusos/lab-test --phase storage --skip-deploy examples/lab-cluster.yaml
-./incusos/lab-test --phase network --skip-deploy examples/lab-cluster.yaml
-./incusos/lab-test --phase projects --skip-deploy examples/lab-cluster.yaml
-./incusos/lab-test --phase backup --skip-deploy examples/lab-cluster.yaml
-./incusos/lab-test --phase limits --skip-deploy examples/lab-cluster.yaml
-./incusos/lab-test --phase security --skip-deploy examples/lab-cluster.yaml
-./incusos/lab-test --phase resilience --skip-deploy examples/lab-cluster.yaml
+bin/lab-test --phase storage --skip-deploy examples/lab-cluster.yaml
+bin/lab-test --phase network --skip-deploy examples/lab-cluster.yaml
+bin/lab-test --phase projects --skip-deploy examples/lab-cluster.yaml
+bin/lab-test --phase backup --skip-deploy examples/lab-cluster.yaml
+bin/lab-test --phase limits --skip-deploy examples/lab-cluster.yaml
+bin/lab-test --phase security --skip-deploy examples/lab-cluster.yaml
+bin/lab-test --phase resilience --skip-deploy examples/lab-cluster.yaml
 ```
 
 ### Extended Test Phases
@@ -660,17 +659,17 @@ and detailed test results.
 
 ```bash
 # Remove test instances only (keeps VMs and cluster)
-./incusos/lab-test --cleanup examples/lab-cluster.yaml
+bin/lab-test --cleanup examples/lab-cluster.yaml
 
 # Destroy VMs only
-./incusos/incusos-proxmox --cleanup examples/lab-cluster.yaml
+bin/incusos-proxmox --cleanup examples/lab-cluster.yaml
 
 # Full cleanup: VMs + deployment ISO + seeds + remotes + cache
-./incusos/incusos-proxmox --cleanup --deep --yes examples/lab-cluster.yaml
+bin/incusos-proxmox --cleanup --deep --yes examples/lab-cluster.yaml
 
 # Pool-wide: destroy ALL managed VMs (no config file needed)
-./incusos/incusos-proxmox --cleanup-all --yes
-./incusos/incusos-proxmox --cleanup-all --deep --yes   # nuke everything
+bin/incusos-proxmox --cleanup-all --yes
+bin/incusos-proxmox --cleanup-all --deep --yes   # nuke everything
 ```
 
 ---
@@ -707,14 +706,14 @@ openssl pkcs12 -export \
     -out ~/.config/incus/client.pfx
 
 # 4. Verify
-./incusos/incusos-proxmox --doctor
+bin/incusos-proxmox --doctor
 ```
 
 ### Step 1: Deploy OC Server
 
 ```bash
-./incusos/incusos-proxmox --yes incusos/examples/lab-oc-deploy.yaml
-./incusos/incusos-proxmox --status incusos/examples/lab-oc-deploy.yaml
+bin/incusos-proxmox --yes bin/examples/lab-oc-deploy.yaml
+bin/incusos-proxmox --status bin/examples/lab-oc-deploy.yaml
 # Note OC_IP from output
 ```
 
@@ -763,9 +762,9 @@ ls -lh /tmp/IncusOS-oc.iso   # ~3.4 GB
 ### Step 6: Deploy Nodes (Hybrid)
 
 ```bash
-./incusos/incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes \
-    incusos/examples/lab-oc-nodes.yaml
-./incusos/incusos-proxmox --status incusos/examples/lab-oc-nodes.yaml
+bin/incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes \
+    bin/examples/lab-oc-nodes.yaml
+bin/incusos-proxmox --status bin/examples/lab-oc-nodes.yaml
 ```
 
 `incusos-proxmox` creates VMs, uploads the OC ISO to ide2, generates per-node
@@ -858,10 +857,10 @@ incus delete oc-node-01:test-ct --force 2>/dev/null
 incus delete oc-node-01:test-vm --force 2>/dev/null
 
 # Nodes
-./incusos/incusos-proxmox --cleanup --deep --yes incusos/examples/lab-oc-nodes.yaml
+bin/incusos-proxmox --cleanup --deep --yes bin/examples/lab-oc-nodes.yaml
 
 # OC server
-./incusos/incusos-proxmox --cleanup --deep --yes incusos/examples/lab-oc-deploy.yaml
+bin/incusos-proxmox --cleanup --deep --yes bin/examples/lab-oc-deploy.yaml
 
 # Remotes
 operations-center remote remove oc-lab 2>/dev/null

bin/deploy-awx

@@ -13,7 +13,7 @@ set -euo pipefail
 readonly VERSION="0.1.0"
 readonly SCRIPT_NAME="deploy-awx"
 readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-readonly MANIFESTS_DIR="${SCRIPT_DIR}/awx-manifests"
+readonly MANIFESTS_DIR="${SCRIPT_DIR}/data/awx-manifests"
 readonly AWX_NAMESPACE="awx"
 
 # ---------------------------------------------------------------------------
@@ -21,24 +21,26 @@ readonly AWX_NAMESPACE="awx"
 # ---------------------------------------------------------------------------
 
 VM_NAME="awx"
-TARGET_REMOTE="oc-node-02"
-AWX_IP="192.168.102.161"
+TARGET_REMOTE=""
+TARGET_NODE=""   # Node within cluster for --target; defaults to TARGET_REMOTE if empty
+AWX_IP=""
 AWX_PREFIX="22"
-AWX_GATEWAY="192.168.100.1"
-AWX_DNS="192.168.100.1"
+AWX_GATEWAY=""
+AWX_DNS=""
 AWX_CPU="4"
 AWX_MEMORY="8GiB"
 AWX_DISK="40GiB"
-GIT_REPO="ssh://git@192.168.1.200:2222/maarten/incus-contrib.git"
+GIT_REPO=""
 GIT_BRANCH="master"
 PLAYBOOK_DIR="playbooks"
-AETHER_URL="https://192.168.102.160:8443"
-AETHER_CLUSTER_ID="52"
+AETHER_URL=""
+AETHER_CLUSTER_ID=""
 
 # Runtime flags
 DRY_RUN=false
 VERBOSE=false
 QUIET=false
+RESUME=false
 CONFIG_FILE=""
 ACTION=""
 
@@ -163,8 +165,10 @@ parse_config() {
     local val
     val=$(grep -A50 '^awx:' "$file" | grep '^ *vm_name:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
     [[ -n "$val" ]] && VM_NAME="$val"
-    val=$(grep -A50 '^awx:' "$file" | grep '^ *target_node:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
+    val=$(grep -A50 '^awx:' "$file" | grep '^ *target_remote:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
     [[ -n "$val" ]] && TARGET_REMOTE="$val"
+    val=$(grep -A50 '^awx:' "$file" | grep '^ *target_node:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
+    [[ -n "$val" ]] && TARGET_NODE="$val"
     val=$(grep -A50 '^awx:' "$file" | grep '^ *ip:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
     if [[ -n "$val" ]]; then
         AWX_IP="${val%%/*}"
@@ -184,7 +188,7 @@ parse_config() {
     [[ -n "$val" ]] && GIT_REPO="$val"
     val=$(grep -A50 '^awx:' "$file" | grep '^ *git_branch:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
     [[ -n "$val" ]] && GIT_BRANCH="$val"
-    val=$(grep -A50 '^awx:' "$file" | grep '^ *aether_url:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
+    val=$(grep -A50 '^awx:' "$file" | grep '^ *aether_url:' | head -1 | sed 's/^ *aether_url: *//' | tr -d '"' || true)
     [[ -n "$val" ]] && AETHER_URL="$val"
     val=$(grep -A50 '^awx:' "$file" | grep '^ *aether_cluster_id:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
     [[ -n "$val" ]] && AETHER_CLUSTER_ID="$val"
@@ -375,8 +379,10 @@ action_deploy() {
     echo
 
     if vm_exists; then
-        if vm_running; then
-            error "AWX VM already exists and is running. Use --status, --heal, or --cleanup first."
+        if [[ "${RESUME:-false}" == true ]]; then
+            info "AWX VM already exists — resuming from phase 4"
+        elif vm_running; then
+            error "AWX VM already exists and is running. Use --status, --heal, --resume, or --cleanup first."
             return 1
         else
             warn "AWX VM exists but is not running — starting it"
@@ -387,9 +393,11 @@ action_deploy() {
         fi
     fi
 
+    if [[ "${RESUME:-false}" != true ]]; then
         phase_create_vm
         phase_configure_network
         phase_install_k3s
+    fi
     phase_deploy_awx_operator
     phase_deploy_awx_instance
     phase_verify
@@ -410,12 +418,13 @@ action_deploy() {
 phase_create_vm() {
     step "Phase 1: Create VM"
 
-    if ! dry_run_guard "incus launch images:debian/12 $(vm_ref) --vm --target ${TARGET_REMOTE} ..."; then
+    local target_node="${TARGET_NODE:-$TARGET_REMOTE}"
+    if ! dry_run_guard "incus launch images:debian/12 $(vm_ref) --vm --target ${target_node} ..."; then
         return 0
     fi
 
     incus launch images:debian/12 "$(vm_ref)" --vm \
-        --target "$TARGET_REMOTE" \
+        --target "$target_node" \
         -c "limits.cpu=${AWX_CPU}" \
         -c "limits.memory=${AWX_MEMORY}" \
         -d "root,size=${AWX_DISK}"
@@ -490,7 +499,7 @@ phase_install_k3s() {
     vm_exec bash -c "
         export DEBIAN_FRONTEND=noninteractive
         apt-get update -qq
-        apt-get install -y -qq curl python3 2>&1 | tail -1
+        apt-get install -y -qq curl python3 git 2>&1 | tail -1
     "
 
     vm_exec bash -c "
@@ -1162,6 +1171,7 @@ parse_args() {
             --cleanup)     ACTION="cleanup";;
             --doctor)      ACTION="doctor";;
             -c|--config)   CONFIG_FILE="$2"; shift;;
+            --resume)      RESUME=true;;
             -n|--dry-run)  DRY_RUN=true;;
             -v|--verbose)  VERBOSE=true;;
             -q|--quiet)    QUIET=true;;
@@ -1186,6 +1196,21 @@ parse_args() {
 # Main
 # ---------------------------------------------------------------------------
 
+require_config() {
+    local missing=()
+    [[ -z "$TARGET_REMOTE" ]]    && missing+=("target_remote")
+    [[ -z "$AWX_IP" ]]           && missing+=("ip")
+    [[ -z "$AWX_GATEWAY" ]]      && missing+=("gateway")
+    [[ -z "$AWX_DNS" ]]          && missing+=("dns")
+    [[ -z "$AETHER_URL" ]]       && missing+=("aether_url")
+    [[ -z "$AETHER_CLUSTER_ID" ]] && missing+=("aether_cluster_id")
+    if [[ ${#missing[@]} -gt 0 ]]; then
+        error "Required config missing: ${missing[*]}"
+        error "Use --config FILE to load environment settings (see envs/)"
+        exit 1
+    fi
+}
+
 main() {
     setup_colors
     parse_args "$@"
@@ -1195,6 +1220,11 @@ main() {
         parse_config "$CONFIG_FILE"
     fi
 
+    # Validate required fields (must come from --config or CLI flags)
+    if [[ "$ACTION" != "doctor" ]]; then
+        require_config
+    fi
+
     echo
     case "$ACTION" in
         deploy)      action_deploy;;

bin/deploy-haproxy

@@ -16,27 +16,26 @@ set -euo pipefail
 readonly VERSION="0.1.0"
 readonly SCRIPT_NAME="deploy-haproxy"
 readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-readonly ENV_FILE="${SCRIPT_DIR}/../env"
 
 # ---------------------------------------------------------------------------
 # Defaults (overridden by config file or CLI flags)
 # ---------------------------------------------------------------------------
 
-CLUSTER_REMOTE="oc-node-01"
-CLUSTER_ID="52"
+CLUSTER_REMOTE=""
+CLUSTER_ID=""
 OVN_NETWORK="net-prod"
-VIP="192.168.103.200"
-HAPROXY_01_IP="10.10.10.50"
-HAPROXY_02_IP="10.10.10.51"
+VIP=""
+HAPROXY_01_IP=""
+HAPROXY_02_IP=""
 HAPROXY_CPU="2"
 HAPROXY_MEMORY="1GB"
-BACKEND_IPS="10.10.10.60,10.10.10.61,10.10.10.62"
+BACKEND_IPS=""
 BACKEND_PORT="80"
 SERVICE_NAME="web-test"
 SERVICE_MODE="http"
 SERVICE_BALANCE="roundrobin"
 IMAGE_VERSION="1.0.0"
-AETHER_URL="https://192.168.102.160:8443"
+AETHER_URL=""
 AETHER_USER="admin"
 AETHER_PASSWORD=""
 
@@ -184,7 +183,7 @@ parse_config() {
     [[ -n "$val" ]] && SERVICE_NAME="$val"
     val=$(grep -A50 '^haproxy:' "$file" | grep '^ *image_version:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
     [[ -n "$val" ]] && IMAGE_VERSION="$val"
-    val=$(grep -A50 '^haproxy:' "$file" | grep '^ *aether_url:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
+    val=$(grep -A50 '^haproxy:' "$file" | grep '^ *aether_url:' | head -1 | sed 's/^ *aether_url: *//' | tr -d '"' || true)
     [[ -n "$val" ]] && AETHER_URL="$val"
 }
 
@@ -197,18 +196,29 @@ load_password() {
         return 0
     fi
 
-    if [[ -f "$ENV_FILE" ]]; then
+    # Check if already exported (user sourced envs/*/env)
+    if [[ -n "${AETHER_ADMIN_PASSWORD:-}" ]]; then
+        AETHER_PASSWORD="$AETHER_ADMIN_PASSWORD"
+        return 0
+    fi
+
+    # Search for env file: walk up from script dir
+    local dir="$SCRIPT_DIR"
+    while [[ "$dir" != "/" ]]; do
+        if [[ -f "${dir}/env" ]]; then
             local val
-        val=$(grep 'AETHER_ADMIN_PASSWORD=' "$ENV_FILE" | head -1 | sed 's/^.*AETHER_ADMIN_PASSWORD=//' | tr -d '"' | tr -d "'" || true)
+            val=$(grep 'AETHER_ADMIN_PASSWORD=' "${dir}/env" | head -1 | sed 's/^.*AETHER_ADMIN_PASSWORD=//' | tr -d '"' | tr -d "'" || true)
             if [[ -n "$val" ]]; then
                 AETHER_PASSWORD="$val"
-            detail "Password loaded from env file"
+                detail "Password loaded from ${dir}/env"
                 return 0
             fi
         fi
+        dir="$(dirname "$dir")"
+    done
 
     error "Aether password not found"
-    error "Set AETHER_ADMIN_PASSWORD in ${ENV_FILE} or use --password"
+    error "Source envs/<env>/env or use --password"
     return 1
 }
 
@@ -1190,6 +1200,22 @@ parse_args() {
 # Main
 # ---------------------------------------------------------------------------
 
+require_config() {
+    local missing=()
+    [[ -z "$CLUSTER_REMOTE" ]] && missing+=("cluster_remote")
+    [[ -z "$CLUSTER_ID" ]]     && missing+=("cluster_id")
+    [[ -z "$VIP" ]]            && missing+=("vip")
+    [[ -z "$HAPROXY_01_IP" ]]  && missing+=("haproxy_01_ip")
+    [[ -z "$HAPROXY_02_IP" ]]  && missing+=("haproxy_02_ip")
+    [[ -z "$BACKEND_IPS" ]]    && missing+=("backend_ips")
+    [[ -z "$AETHER_URL" ]]     && missing+=("aether_url")
+    if [[ ${#missing[@]} -gt 0 ]]; then
+        error "Required config missing: ${missing[*]}"
+        error "Use --config FILE to load environment settings (see envs/)"
+        exit 1
+    fi
+}
+
 main() {
     setup_colors
     parse_args "$@"
@@ -1199,6 +1225,11 @@ main() {
         parse_config "$CONFIG_FILE"
     fi
 
+    # Validate required fields (must come from --config or CLI flags)
+    if [[ "$ACTION" != "doctor" ]]; then
+        require_config
+    fi
+
     echo
     case "$ACTION" in
         deploy)  action_deploy;;

bin/deploy-observability

@@ -14,35 +14,37 @@ set -euo pipefail
 readonly VERSION="0.1.0"
 readonly SCRIPT_NAME="deploy-observability"
 readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-readonly DASHBOARDS_DIR="${SCRIPT_DIR}/observability-dashboards"
+readonly DASHBOARDS_DIR="${SCRIPT_DIR}/data/observability-dashboards"
 
 # ---------------------------------------------------------------------------
-# Defaults (overridden by CLI flags)
+# Defaults (overridden by config file or CLI flags)
 # ---------------------------------------------------------------------------
 
-CLUSTER_REMOTE="oc-node-01"
+CLUSTER_REMOTE=""
 OVN_NETWORK="net-prod"
 MONITORING_IP="10.10.10.70"
-MONITORING_TARGET="oc-node-02"
-FORWARD_VIP="192.168.103.201"
+MONITORING_TARGET=""
+FORWARD_VIP=""
 
 # Fallback defaults — used when cluster is unreachable (e.g., --dry-run)
-DEFAULT_INCUS_NODES=("192.168.102.140" "192.168.102.141" "192.168.102.142")
-DEFAULT_NODE_EXP_TARGETS=("oc-node-01" "oc-node-02" "oc-node-03")
-DEFAULT_NODE_EXP_IPS=("10.10.10.71" "10.10.10.72" "10.10.10.73")
+DEFAULT_INCUS_NODES=()
+DEFAULT_NODE_EXP_TARGETS=()
+DEFAULT_NODE_EXP_IPS=()
 
 INCUS_NODES=()
 NODE_EXP_TARGETS=()
 NODE_EXP_IPS=()
-HAPROXY_IPS=("10.10.10.50" "10.10.10.51")
+HAPROXY_IPS=()
 
-HAPROXY_CLUSTER_ID="52"
+HAPROXY_CLUSTER_ID=""
 OC_SERVER_IP=""
 
 # Runtime flags
 DRY_RUN=false
 VERBOSE=false
 QUIET=false
+RESUME=false
+CONFIG_FILE=""
 ACTION=""
 MANUAL_NODES=""
 
@@ -101,8 +103,12 @@ ${BOLD}ACTIONS${RESET}
     --doctor          Diagnose common issues
 
 ${BOLD}OPTIONS${RESET}
+    -c, --config FILE         Load environment config from YAML file (see envs/)
     -r, --remote NAME         Incus remote (default: ${CLUSTER_REMOTE})
     --nodes NODE1,NODE2       Manual node list (skip auto-discovery)
+    --monitoring-target NODE  Node to deploy monitoring container on (default: ${MONITORING_TARGET})
+    --monitoring-ip IP        IP for monitoring container (default: ${MONITORING_IP})
+    --forward-vip IP          OVN external IP for Grafana/Prometheus forward (default: ${FORWARD_VIP})
     --oc-server IP            Monitor oc-server at this IP (proxy device for 9100)
     -n, --dry-run         Preview actions without executing
     -v, --verbose         Show detailed output
@@ -298,16 +304,22 @@ action_deploy() {
     echo
 
     if container_exists "monitoring"; then
+        if [[ "$RESUME" == true ]]; then
+            info "Monitoring container already exists — resuming from phase 5b"
+        else
             error "Monitoring container already exists"
-        error "Use --status to check health, or --cleanup first."
+            error "Use --status to check health, --cleanup first, or --resume to continue."
             return 1
         fi
+    fi
 
+    if [[ "$RESUME" != true ]]; then
         phase_deploy_monitoring
         phase_install_prometheus
         phase_install_loki
         phase_install_grafana
         phase_deploy_node_exporters
+    fi
     phase_deploy_oc_server_exporter
     phase_create_acl
     phase_configure_haproxy_exporter
@@ -1694,6 +1706,48 @@ except Exception as e:
     fi
 }
 
+# ---------------------------------------------------------------------------
+# Config file parsing
+# ---------------------------------------------------------------------------
+
+parse_config() {
+    local file="$1"
+    if [[ ! -f "$file" ]]; then
+        error "Config file not found: $file"
+        exit 1
+    fi
+
+    local val
+    val=$(grep -A50 '^observability:' "$file" | grep '^ *cluster_remote:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
+    [[ -n "$val" ]] && CLUSTER_REMOTE="$val"
+    val=$(grep -A50 '^observability:' "$file" | grep '^ *ovn_network:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
+    [[ -n "$val" ]] && OVN_NETWORK="$val"
+    val=$(grep -A50 '^observability:' "$file" | grep '^ *monitoring_ip:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
+    [[ -n "$val" ]] && MONITORING_IP="$val"
+    val=$(grep -A50 '^observability:' "$file" | grep '^ *monitoring_target:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
+    [[ -n "$val" ]] && MONITORING_TARGET="$val"
+    val=$(grep -A50 '^observability:' "$file" | grep '^ *forward_vip:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
+    [[ -n "$val" ]] && FORWARD_VIP="$val"
+    val=$(grep -A50 '^observability:' "$file" | grep '^ *incus_nodes:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
+    if [[ -n "$val" ]]; then
+        IFS=',' read -ra DEFAULT_INCUS_NODES <<< "$val"
+    fi
+    val=$(grep -A50 '^observability:' "$file" | grep '^ *node_exp_targets:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
+    if [[ -n "$val" ]]; then
+        IFS=',' read -ra DEFAULT_NODE_EXP_TARGETS <<< "$val"
+    fi
+    val=$(grep -A50 '^observability:' "$file" | grep '^ *node_exp_ips:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
+    if [[ -n "$val" ]]; then
+        IFS=',' read -ra DEFAULT_NODE_EXP_IPS <<< "$val"
+    fi
+    val=$(grep -A50 '^observability:' "$file" | grep '^ *haproxy_ips:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
+    if [[ -n "$val" ]]; then
+        IFS=',' read -ra HAPROXY_IPS <<< "$val"
+    fi
+    val=$(grep -A50 '^observability:' "$file" | grep '^ *haproxy_cluster_id:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
+    [[ -n "$val" ]] && HAPROXY_CLUSTER_ID="$val"
+}
+
 # ---------------------------------------------------------------------------
 # Argument parsing
 # ---------------------------------------------------------------------------
@@ -1706,9 +1760,14 @@ parse_args() {
             --status)      ACTION="status";;
             --cleanup)     ACTION="cleanup";;
             --doctor)      ACTION="doctor";;
+            -c|--config)           CONFIG_FILE="$2"; shift;;
             -r|--remote)           CLUSTER_REMOTE="$2"; shift;;
             --nodes)               MANUAL_NODES="$2"; shift;;
+            --monitoring-target)   MONITORING_TARGET="$2"; shift;;
+            --monitoring-ip)       MONITORING_IP="$2"; shift;;
+            --forward-vip)         FORWARD_VIP="$2"; shift;;
             --oc-server)           OC_SERVER_IP="$2"; shift;;
+            --resume)              RESUME=true;;
             -n|--dry-run)  DRY_RUN=true;;
             -v|--verbose)  VERBOSE=true;;
             -q|--quiet)    QUIET=true;;
@@ -1729,6 +1788,18 @@ parse_args() {
     fi
 }
 
+require_config() {
+    local missing=()
+    [[ -z "$CLUSTER_REMOTE" ]]    && missing+=("cluster_remote")
+    [[ -z "$FORWARD_VIP" ]]       && missing+=("forward_vip")
+    [[ -z "$MONITORING_TARGET" ]] && missing+=("monitoring_target")
+    if [[ ${#missing[@]} -gt 0 ]]; then
+        error "Required config missing: ${missing[*]}"
+        error "Use --config FILE to load environment settings (see envs/)"
+        exit 1
+    fi
+}
+
 # ---------------------------------------------------------------------------
 # Main
 # ---------------------------------------------------------------------------
@@ -1737,6 +1808,16 @@ main() {
     setup_colors
     parse_args "$@"
 
+    # Load config file if provided
+    if [[ -n "$CONFIG_FILE" ]]; then
+        parse_config "$CONFIG_FILE"
+    fi
+
+    # Validate required fields (must come from --config or CLI flags)
+    if [[ "$ACTION" != "doctor" ]]; then
+        require_config
+    fi
+
     # Discover cluster nodes (populates INCUS_NODES, NODE_EXP_TARGETS, NODE_EXP_IPS)
     discover_cluster_nodes
 

bin/diagnostic/investigate-boot

@@ -163,6 +163,20 @@ done
 # ---------------------------------------------------------------------------
 
 load_env() {
+    [[ -n "${PROXMOX_TOKEN_SECRET:-}" ]] && return
+
+    # Check envs/*/env in repo root
+    local repo_root
+    repo_root=$(cd "$SCRIPT_DIR/../.." && pwd)
+    for env_file in "${repo_root}"/envs/*/env; do
+        if [[ -f "$env_file" ]]; then
+            # shellcheck disable=SC1091
+            source "$env_file"
+            [[ -n "${PROXMOX_TOKEN_SECRET:-}" ]] && return
+        fi
+    done
+
+    # Walk up from script dir (backward compat)
     local dir="$SCRIPT_DIR"
     while [[ "$dir" != "/" ]]; do
         if [[ -f "${dir}/env" ]]; then

bin/diagnostic/investigate-parallel

@@ -86,6 +86,16 @@ done
 # ---------------------------------------------------------------------------
 
 load_env() {
+    [[ -n "${PROXMOX_TOKEN_SECRET:-}" ]] && return
+
+    # Check envs/*/env in repo root
+    local repo_root
+    repo_root=$(cd "$SCRIPT_DIR/../.." && pwd)
+    for env_file in "${repo_root}"/envs/*/env; do
+        [[ -f "$env_file" ]] && { source "$env_file"; [[ -n "${PROXMOX_TOKEN_SECRET:-}" ]] && return; }
+    done
+
+    # Walk up from script dir (backward compat)
     local dir="$SCRIPT_DIR"
     while [[ "$dir" != "/" ]]; do
         [[ -f "${dir}/env" ]] && { source "${dir}/env"; return; }
@@ -97,9 +107,21 @@ load_env
 [[ -z "${PROXMOX_TOKEN_SECRET:-}" ]] && { error "PROXMOX_TOKEN_SECRET not set"; exit 1; }
 [[ -z "${PROXMOX_ROOT_PASSWORD:-}" ]] && { error "PROXMOX_ROOT_PASSWORD not set"; exit 1; }
 
+find_proxmox_yaml() {
+    local repo_root
+    repo_root=$(cd "$SCRIPT_DIR/../.." && pwd)
+    for f in "${repo_root}"/envs/*/proxmox.yaml; do
+        [[ -f "$f" ]] && { echo "$f"; return; }
+    done
+    [[ -f "${repo_root}/proxmox.yaml" ]] && { echo "${repo_root}/proxmox.yaml"; return; }
+    echo ""
+}
+PROXMOX_YAML=$(find_proxmox_yaml)
+[[ -z "$PROXMOX_YAML" ]] && { error "proxmox.yaml not found"; exit 1; }
+
 yaml_get() {
     local val
-    val=$(awk -F': ' -v k="$1" '$1 == k {print $2; exit}' "${SCRIPT_DIR}/proxmox.yaml" 2>/dev/null) || val=""
+    val=$(awk -F': ' -v k="$1" '$1 == k {print $2; exit}' "$PROXMOX_YAML" 2>/dev/null) || val=""
     val="${val#\"}" ; val="${val%\"}" ; val="${val#\'}" ; val="${val%\'}"
     echo "${val:-${2:-}}"
 }
@@ -448,7 +470,7 @@ for ((i=0; i<COUNT; i++)); do
 import subprocess, os, sys
 # Read token_id from proxmox.yaml
 token_id = ''
-yaml_file = '${SCRIPT_DIR}/proxmox.yaml'
+yaml_file = '${PROXMOX_YAML}'
 for line in open(yaml_file):
     line = line.strip()
     if line.startswith('api_token_id:'):

bin/diagnostic/observe-deploy

@@ -30,7 +30,8 @@ set -euo pipefail
 readonly VERSION="0.1.0"
 readonly SCRIPT_NAME="observe-deploy"
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-SEED_TOOL="${SCRIPT_DIR}/incusos-seed"
+BIN_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
+SEED_TOOL="${BIN_DIR}/incusos-seed"
 
 # ---------------------------------------------------------------------------
 # Defaults
@@ -184,6 +185,20 @@ done
 # ---------------------------------------------------------------------------
 
 load_env() {
+    [[ -n "${PROXMOX_TOKEN_SECRET:-}" ]] && return
+
+    # Check envs/*/env in repo root
+    local repo_root
+    repo_root=$(cd "$SCRIPT_DIR/../.." && pwd)
+    for env_file in "${repo_root}"/envs/*/env; do
+        if [[ -f "$env_file" ]]; then
+            # shellcheck disable=SC1091
+            source "$env_file"
+            [[ -n "${PROXMOX_TOKEN_SECRET:-}" ]] && return
+        fi
+    done
+
+    # Walk up from script dir (backward compat)
     local dir="$SCRIPT_DIR"
     while [[ "$dir" != "/" ]]; do
         if [[ -f "${dir}/env" ]]; then
@@ -222,14 +237,28 @@ yaml_get() {
 
 load_proxmox_yaml() {
     local yaml_file=""
-    if [[ -f "${SCRIPT_DIR}/proxmox.yaml" ]]; then
-        yaml_file="${SCRIPT_DIR}/proxmox.yaml"
+    local repo_root
+    repo_root=$(cd "$SCRIPT_DIR/../.." && pwd)
+
+    # Check envs/*/proxmox.yaml
+    for f in "${repo_root}"/envs/*/proxmox.yaml; do
+        if [[ -f "$f" ]]; then
+            yaml_file="$f"
+            break
+        fi
+    done
+
+    # Backward compat: repo root or cwd
+    if [[ -z "$yaml_file" ]]; then
+        if [[ -f "${repo_root}/proxmox.yaml" ]]; then
+            yaml_file="${repo_root}/proxmox.yaml"
         elif [[ -f "proxmox.yaml" ]]; then
             yaml_file="proxmox.yaml"
         else
             error "proxmox.yaml not found"
             exit 1
         fi
+    fi
 
     PVE_HOST=$(yaml_get "$yaml_file" "host")
     PVE_NODE=$(yaml_get "$yaml_file" "node" "pve")
@@ -1026,7 +1055,7 @@ setup_output() {
         fi
     fi
 
-    RUN_DIR="${SCRIPT_DIR}/observe-runs/${timestamp}_${LABEL}"
+    RUN_DIR="${BIN_DIR}/diagnostic/observe-runs/${timestamp}_${LABEL}"
     mkdir -p "$RUN_DIR"
 
     LOG_FILE="${RUN_DIR}/deploy.log"
@@ -1315,7 +1344,7 @@ else
     echo ""
 
     # Output directories
-    echo "  Output: ${SCRIPT_DIR}/observe-runs/*${BASE_LABEL}*"
+    echo "  Output: ${BIN_DIR}/diagnostic/observe-runs/*${BASE_LABEL}*"
     echo ""
 
     if [[ $total_fail -gt 0 ]]; then

bin/diagnostic/test-boot-reliability

@@ -97,8 +97,21 @@ done
 # Load environment
 # ---------------------------------------------------------------------------
 
-# Auto-load env file for PROXMOX_TOKEN_SECRET
 load_env() {
+    [[ -n "${PROXMOX_TOKEN_SECRET:-}" ]] && return
+
+    # Check envs/*/env in repo root
+    local repo_root
+    repo_root=$(cd "$SCRIPT_DIR/../.." && pwd)
+    for env_file in "${repo_root}"/envs/*/env; do
+        if [[ -f "$env_file" ]]; then
+            # shellcheck disable=SC1091
+            source "$env_file"
+            [[ -n "${PROXMOX_TOKEN_SECRET:-}" ]] && return
+        fi
+    done
+
+    # Walk up from script dir (backward compat)
     local dir="$SCRIPT_DIR"
     while [[ "$dir" != "/" ]]; do
         if [[ -f "${dir}/env" ]]; then

bin/examples/proxmox.yaml.example

@@ -1,7 +1,7 @@
 # proxmox.yaml.example - Proxmox connection config template
 #
-# Copy to incusos/proxmox.yaml and edit with your settings.
-# This file is auto-discovered by incusos-proxmox and lab-test.
+# Copy to envs/<env>/proxmox.yaml and edit with your settings.
+# Use with: bin/incusos-proxmox --proxmox envs/<env>/proxmox.yaml
 #
 # Lab config files (lab-cluster.yaml, lab-oc.yaml, etc.) reference
 # this for connection settings, so you only need to change credentials

bin/helpers/proxmox-api

@@ -78,6 +78,21 @@ EXTRA_ARGS=("${ARGS[@]:2}")
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 
 load_env() {
+    # Already exported (user sourced envs/*/env)
+    [[ -n "${PROXMOX_TOKEN_SECRET:-}" ]] && return
+
+    # Check envs/*/env in repo root
+    local repo_root
+    repo_root=$(cd "$SCRIPT_DIR/../.." && pwd)
+    for env_file in "${repo_root}"/envs/*/env; do
+        if [[ -f "$env_file" ]]; then
+            # shellcheck disable=SC1091
+            source "$env_file"
+            [[ -n "${PROXMOX_TOKEN_SECRET:-}" ]] && return
+        fi
+    done
+
+    # Walk up from script dir (backward compat)
     local dir="$SCRIPT_DIR"
     while [[ "$dir" != "/" ]]; do
         if [[ -f "${dir}/env" ]]; then
@@ -109,9 +124,21 @@ yaml_get() {
 }
 
 find_proxmox_yaml() {
+    local repo_root
+    repo_root=$(cd "$SCRIPT_DIR/../.." && pwd)
+
+    # Check envs/*/proxmox.yaml in repo root
+    for f in "${repo_root}"/envs/*/proxmox.yaml; do
+        if [[ -f "$f" ]]; then
+            echo "$f"
+            return
+        fi
+    done
+
+    # Walk up from script dir (backward compat)
     local search_dirs=(
         "${SCRIPT_DIR}/.."
-        "${SCRIPT_DIR}/../.."
+        "${repo_root}"
         "$(pwd)"
     )
     for dir in "${search_dirs[@]}"; do
@@ -119,10 +146,6 @@ find_proxmox_yaml() {
             echo "${dir}/proxmox.yaml"
             return
         fi
-        if [[ -f "${dir}/incusos/proxmox.yaml" ]]; then
-            echo "${dir}/incusos/proxmox.yaml"
-            return
-        fi
     done
     echo ""
 }

bin/helpers/proxmox-screenshot

@@ -71,6 +71,21 @@ fi
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 
 load_env() {
+    # Already exported (user sourced envs/*/env)
+    [[ -n "${PROXMOX_ROOT_PASSWORD:-}" ]] && return
+
+    # Check envs/*/env in repo root
+    local repo_root
+    repo_root=$(cd "$SCRIPT_DIR/../.." && pwd)
+    for env_file in "${repo_root}"/envs/*/env; do
+        if [[ -f "$env_file" ]]; then
+            # shellcheck disable=SC1091
+            source "$env_file"
+            [[ -n "${PROXMOX_ROOT_PASSWORD:-}" ]] && return
+        fi
+    done
+
+    # Walk up from script dir (backward compat)
     local dir="$SCRIPT_DIR"
     while [[ "$dir" != "/" ]]; do
         if [[ -f "${dir}/env" ]]; then
@@ -102,10 +117,21 @@ yaml_get() {
 }
 
 find_proxmox_yaml() {
-    # Search: script's grandparent (repo/incusos -> repo), script parent, cwd
+    local repo_root
+    repo_root=$(cd "$SCRIPT_DIR/../.." && pwd)
+
+    # Check envs/*/proxmox.yaml in repo root
+    for f in "${repo_root}"/envs/*/proxmox.yaml; do
+        if [[ -f "$f" ]]; then
+            echo "$f"
+            return
+        fi
+    done
+
+    # Walk up from script dir (backward compat)
     local search_dirs=(
         "${SCRIPT_DIR}/.."
-        "${SCRIPT_DIR}/../.."
+        "${repo_root}"
         "$(pwd)"
     )
     for dir in "${search_dirs[@]}"; do
@@ -113,10 +139,6 @@ find_proxmox_yaml() {
             echo "${dir}/proxmox.yaml"
             return
         fi
-        if [[ -f "${dir}/incusos/proxmox.yaml" ]]; then
-            echo "${dir}/incusos/proxmox.yaml"
-            return
-        fi
     done
     echo ""
 }

bin/incusos-proxmox

@@ -22,10 +22,10 @@ readonly MANAGED_MARKER="[incusos-lab:managed]"
 # ---------------------------------------------------------------------------
 # Auto-load env file (secrets like PROXMOX_TOKEN_SECRET)
 # ---------------------------------------------------------------------------
-# Searches for an 'env' file: first next to the script, then walking up from
-# the current directory. Sources it if found and not already loaded. This is
-# a convenience -- users can still export variables manually or source the
-# file themselves.
+# Searches for an 'env' file. Sources it if found and not already loaded.
+# Priority: already exported > envs/*/env in repo > walk up from cwd.
+# This is a convenience -- users can still export variables manually or
+# source the file themselves (e.g., source envs/hetzner/env).
 
 load_env_file() {
     # Skip if the secret is already set (user exported it, or env already sourced)
@@ -35,18 +35,24 @@ load_env_file() {
 
     local script_real
     script_real=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
+    local repo_root
+    repo_root=$(cd "$script_real/.." && pwd)
 
-    # Check: script directory's parent (repo root when script is in incusos/)
-    if [[ -f "${script_real}/../env" ]]; then
+    # Check: envs/*/env in repo root (per-environment secrets)
+    for env_file in "${repo_root}"/envs/*/env; do
+        if [[ -f "$env_file" ]]; then
             # shellcheck source=/dev/null
-        source "${script_real}/../env"
+            source "$env_file"
+            if [[ -n "${PROXMOX_TOKEN_SECRET:-}" ]]; then
                 return 0
             fi
+        fi
+    done
 
-    # Check: script directory itself
-    if [[ -f "${script_real}/env" ]]; then
+    # Check: repo root env file (backward compat)
+    if [[ -f "${repo_root}/env" ]]; then
         # shellcheck source=/dev/null
-        source "${script_real}/env"
+        source "${repo_root}/env"
         return 0
     fi
 
@@ -1405,6 +1411,13 @@ json.dump(result, sys.stdout, indent=2)
     val=$(json_get "$px_json" ".api_token_id" ""); [[ -n "$val" ]] && PVE_API_TOKEN_ID="$val"
     val=$(json_get "$px_json" ".pool" ""); [[ -n "$val" ]] && PVE_POOL="$val"
     val=$(json_get "$px_json" ".method" ""); [[ -n "$val" ]] && PVE_METHOD="$val"
+    # api_token_secret in proxmox.yaml overrides the repo-wide env file, useful
+    # for per-target configs where the env file holds a different target's token.
+    val=$(json_get "$px_json" ".api_token_secret" "")
+    if [[ -n "$val" ]]; then
+        export PROXMOX_TOKEN_SECRET="$val"
+        detail "Using api_token_secret from proxmox.yaml"
+    fi
 
     rm -f "$px_json"
 }
@@ -2281,6 +2294,7 @@ build_vm_description() {
 # Check if a VM has our management marker in its description
 pve_vm_is_managed() {
     local vmid="$1"
+    local encoded_marker="${MANAGED_MARKER//:/%3A}"
 
     if [[ "$DRY_RUN" == true ]]; then
         return 1
@@ -2296,7 +2310,8 @@ pve_vm_is_managed() {
             ;;
     esac
 
-    echo "$config_output" | grep -qF "$MANAGED_MARKER"
+    echo "$config_output" | grep -qF "$MANAGED_MARKER" || \
+        echo "$config_output" | grep -qF "$encoded_marker"
 }
 
 # Check if a VM still has install media (ide2) attached.
@@ -2397,15 +2412,15 @@ preflight_checks() {
     if [[ "$DRY_RUN" != true ]]; then
         case "$PVE_METHOD" in
             ssh)
-                if ! pve_run "pvesm status 2>/dev/null" | awk 'NR>1 {print \$1}' | grep -qw "$PVE_STORAGE"; then
+                if ! pve_run "pvesm status 2>/dev/null" | awk 'NR>1 {print $1}' | grep -qw "$PVE_STORAGE"; then
                     error "Storage pool '${PVE_STORAGE}' not found on Proxmox"
-                    error "Available pools: $(pve_run "pvesm status 2>/dev/null" | awk 'NR>1 {print \$1}' | tr '\n' ' ')"
+                    error "Available pools: $(pve_run "pvesm status 2>/dev/null" | awk 'NR>1 {print $1}' | tr '\n' ' ')"
                     error "Fix: update 'proxmox.storage' in your config file"
                     errors=$((errors + 1))
                 fi
-                if ! pve_run "pvesm status 2>/dev/null" | awk 'NR>1 {print \$1}' | grep -qw "$PVE_ISO_STORAGE"; then
+                if ! pve_run "pvesm status 2>/dev/null" | awk 'NR>1 {print $1}' | grep -qw "$PVE_ISO_STORAGE"; then
                     error "ISO storage '${PVE_ISO_STORAGE}' not found on Proxmox"
-                    error "Available pools: $(pve_run "pvesm status 2>/dev/null" | awk 'NR>1 {print \$1}' | tr '\n' ' ')"
+                    error "Available pools: $(pve_run "pvesm status 2>/dev/null" | awk 'NR>1 {print $1}' | tr '\n' ' ')"
                     error "Fix: update 'proxmox.iso_storage' in your config file"
                     errors=$((errors + 1))
                 fi

bin/lab-status

@@ -0,0 +1,436 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+readonly VERSION="0.1.0"
+readonly SCRIPT_NAME="lab-status"
+readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+readonly REPO_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
+readonly ENVS_DIR="${REPO_DIR}/envs"
+
+# --- Defaults -----------------------------------------------------------------
+
+QUIET=false
+VERBOSE=false
+ENV_NAME=""
+NO_CHECK=false
+JSON_OUTPUT=false
+
+# Service arrays (parallel indexed)
+declare -a SVC_NAMES=()
+declare -a SVC_URLS=()
+declare -a SVC_STATUSES=()
+
+# --- Colors & logging --------------------------------------------------------
+
+setup_colors() {
+    if [[ -t 1 ]] && [[ "${NO_COLOR:-}" != "1" ]] && [[ "${TERM:-}" != "dumb" ]]; then
+        RED='\033[0;31m'
+        GREEN='\033[0;32m'
+        YELLOW='\033[0;33m'
+        BLUE='\033[0;34m'
+        CYAN='\033[0;36m'
+        BOLD='\033[1m'
+        DIM='\033[2m'
+        RESET='\033[0m'
+    else
+        RED='' GREEN='' YELLOW='' BLUE='' CYAN='' BOLD='' DIM='' RESET=''
+    fi
+}
+
+info()    { [[ "$QUIET" == true ]] && return; echo -e "${BLUE}[info]${RESET}    $*"; }
+success() { [[ "$QUIET" == true ]] && return; echo -e "${GREEN}[ok]${RESET}      $*"; }
+warn()    { echo -e "${YELLOW}[warn]${RESET}    $*" >&2; }
+error()   { echo -e "${RED}[error]${RESET}   $*" >&2; }
+detail()  { [[ "$VERBOSE" != true ]] && return; echo -e "${DIM}          $*${RESET}"; }
+
+# --- Usage --------------------------------------------------------------------
+
+usage() {
+    cat <<EOF
+${BOLD}${SCRIPT_NAME}${RESET} v${VERSION} - Lab service status overview
+
+${BOLD}USAGE${RESET}
+    ${SCRIPT_NAME} [OPTIONS]
+
+${BOLD}OPTIONS${RESET}
+    -e, --env NAME        Environment name (auto-detects if only one exists)
+        --no-check        Skip reachability checks, just show URLs
+        --json            Machine-readable JSON output
+    -v, --verbose         Show extra details
+    -q, --quiet           Suppress info messages
+    -h, --help            Show this help
+
+${BOLD}DESCRIPTION${RESET}
+    Reads YAML configs from envs/<env>/ and checks whether each service
+    is reachable. Any HTTP response (even 401/403) counts as UP; connection
+    timeout or refused counts as DOWN.
+
+    Services detected: Proxmox, OC Server, Cluster Nodes, Aether, AWX,
+    HAProxy VIP, Grafana, Prometheus.
+
+${BOLD}EXAMPLES${RESET}
+    # Show hetzner lab with reachability checks
+    ${SCRIPT_NAME} --env hetzner
+
+    # Show beelink lab without checks
+    ${SCRIPT_NAME} --env beelink --no-check
+
+    # JSON output for scripting
+    ${SCRIPT_NAME} --env hetzner --json
+
+EOF
+}
+
+# --- YAML helpers -------------------------------------------------------------
+
+# Extract a value from a YAML section: yaml_val FILE SECTION KEY
+yaml_val() {
+    local file="$1" section="$2" key="$3"
+    grep -A50 "^${section}:" "$file" \
+        | grep "^ *${key}:" | head -1 \
+        | sed "s/^ *${key}: *//" | tr -d '"' | sed 's/ *#.*//' || true
+}
+
+# Extract a top-level YAML value: yaml_val_top FILE KEY
+yaml_val_top() {
+    local file="$1" key="$2"
+    grep "^${key}:" "$file" | head -1 \
+        | sed "s/^${key}: *//" | tr -d '"' | sed 's/ *#.*//' || true
+}
+
+# --- Service collection -------------------------------------------------------
+
+add_service() {
+    local name="$1" url="$2"
+    SVC_NAMES+=("$name")
+    SVC_URLS+=("$url")
+    SVC_STATUSES+=("")
+}
+
+detect_env() {
+    if [[ -n "$ENV_NAME" ]]; then
+        if [[ ! -d "${ENVS_DIR}/${ENV_NAME}" ]]; then
+            error "Environment not found: ${ENV_NAME}"
+            local available
+            available=$(ls "$ENVS_DIR" 2>/dev/null | tr '\n' ' ')
+            [[ -n "$available" ]] && error "Available: ${available}"
+            exit 1
+        fi
+        return
+    fi
+
+    local envs=()
+    for d in "$ENVS_DIR"/*/; do
+        [[ -d "$d" ]] && envs+=("$(basename "$d")")
+    done
+
+    if [[ ${#envs[@]} -eq 1 ]]; then
+        ENV_NAME="${envs[0]}"
+        info "Auto-detected environment: ${ENV_NAME}"
+    elif [[ ${#envs[@]} -eq 0 ]]; then
+        error "No environments found in ${ENVS_DIR}/"
+        exit 1
+    else
+        error "Multiple environments found: ${envs[*]}"
+        error "Use --env NAME to select one"
+        exit 1
+    fi
+}
+
+collect_proxmox() {
+    local env_dir="${ENVS_DIR}/${ENV_NAME}"
+    local pve_file=""
+
+    if [[ -f "${env_dir}/proxmox.yaml" ]]; then
+        pve_file="${env_dir}/proxmox.yaml"
+    elif [[ -f "${env_dir}/proxmox.yaml.example" ]]; then
+        pve_file="${env_dir}/proxmox.yaml.example"
+        warn "Using proxmox.yaml.example (proxmox.yaml not found)"
+    else
+        detail "No proxmox.yaml found, skipping Proxmox"
+        return
+    fi
+
+    local host
+    host=$(yaml_val_top "$pve_file" "host")
+    if [[ -n "$host" ]]; then
+        host="${host%%/*}"
+        add_service "Proxmox" "https://${host}:8006"
+        detail "Proxmox from $(basename "$pve_file"): ${host}"
+    fi
+}
+
+collect_lab_vms() {
+    local env_dir="${ENVS_DIR}/${ENV_NAME}"
+    local lab_file=""
+
+    # Prefer lab-production.yaml over lab-cluster.yaml
+    if [[ -f "${env_dir}/lab-production.yaml" ]]; then
+        lab_file="${env_dir}/lab-production.yaml"
+    elif [[ -f "${env_dir}/lab-cluster.yaml" ]]; then
+        lab_file="${env_dir}/lab-cluster.yaml"
+    else
+        detail "No lab-*.yaml found, skipping cluster VMs"
+        return
+    fi
+
+    detail "Lab VMs from $(basename "$lab_file")"
+
+    local vm_data
+    vm_data=$(python3 -c "
+import sys, re
+
+try:
+    import yaml
+    with open(sys.argv[1]) as f:
+        data = yaml.safe_load(f)
+except ImportError:
+    data = {'vms': []}
+    with open(sys.argv[1]) as f:
+        content = f.read()
+    in_vms = False
+    current = {}
+    for line in content.split('\n'):
+        if line.startswith('vms:'):
+            in_vms = True
+            continue
+        if in_vms:
+            if line.startswith('  - '):
+                if current:
+                    data['vms'].append(current)
+                current = {}
+                line = line[4:]
+            elif line.startswith('    '):
+                line = line[4:]
+            elif line and not line.startswith(' ') and not line.startswith('#'):
+                if current:
+                    data['vms'].append(current)
+                break
+            else:
+                continue
+            m = re.match(r'(\w+):\s*(.*)', line.strip())
+            if m:
+                current[m.group(1)] = m.group(2).strip('\"')
+    if current:
+        data['vms'].append(current)
+
+for vm in data.get('vms', []):
+    name = vm.get('name', '')
+    app = vm.get('app', '')
+    ip = vm.get('ip', '').split('/')[0]
+    print(f'{name}|{app}|{ip}')
+" "$lab_file" 2>/dev/null) || return
+
+    local node_idx=0
+    while IFS='|' read -r name app ip; do
+        [[ -z "$ip" ]] && continue
+        case "$app" in
+            operations-center)
+                add_service "OC Server" "https://${ip}:8443"
+                ;;
+            incus)
+                node_idx=$((node_idx + 1))
+                local label
+                label=$(printf "Cluster Node %02d" "$node_idx")
+                add_service "$label" "https://${ip}:8443"
+                ;;
+        esac
+    done <<< "$vm_data"
+}
+
+collect_haproxy() {
+    local env_dir="${ENVS_DIR}/${ENV_NAME}"
+    local file="${env_dir}/haproxy.yaml"
+    [[ -f "$file" ]] || return
+
+    local aether_url vip
+    aether_url=$(yaml_val "$file" "haproxy" "aether_url")
+    vip=$(yaml_val "$file" "haproxy" "vip")
+
+    if [[ -n "$aether_url" ]]; then
+        add_service "Aether" "$aether_url"
+        detail "Aether from haproxy.yaml: ${aether_url}"
+    fi
+    if [[ -n "$vip" ]]; then
+        vip="${vip%%/*}"
+        add_service "HAProxy VIP" "http://${vip}:80"
+        detail "HAProxy VIP from haproxy.yaml: ${vip}"
+    fi
+}
+
+collect_awx() {
+    local env_dir="${ENVS_DIR}/${ENV_NAME}"
+    local file="${env_dir}/awx.yaml"
+    [[ -f "$file" ]] || return
+
+    local ip
+    ip=$(yaml_val "$file" "awx" "ip")
+    [[ -z "$ip" ]] && return
+    ip="${ip%%/*}"
+    add_service "AWX" "http://${ip}:30080"
+    detail "AWX from awx.yaml: ${ip}"
+}
+
+collect_observability() {
+    local env_dir="${ENVS_DIR}/${ENV_NAME}"
+    local file="${env_dir}/observability.yaml"
+    [[ -f "$file" ]] || return
+
+    local forward_vip
+    forward_vip=$(yaml_val "$file" "observability" "forward_vip")
+    [[ -z "$forward_vip" ]] && return
+    forward_vip="${forward_vip%%/*}"
+
+    add_service "Grafana" "http://${forward_vip}:3000"
+    add_service "Prometheus" "http://${forward_vip}:9090"
+    detail "Observability from observability.yaml: forward_vip=${forward_vip}"
+}
+
+# --- Reachability checks -----------------------------------------------------
+
+check_reachability() {
+    local url="$1" result_file="$2"
+    local http_code
+    http_code=$(curl -sk --connect-timeout 2 --max-time 3 \
+        -o /dev/null -w '%{http_code}' "$url" 2>/dev/null) || http_code="000"
+    if [[ "$http_code" != "000" ]]; then
+        echo "UP" > "$result_file"
+    else
+        echo "DOWN" > "$result_file"
+    fi
+}
+
+run_checks() {
+    local tmpdir
+    tmpdir=$(mktemp -d "${TMPDIR:-/tmp}/${SCRIPT_NAME}-XXXXXX")
+    # shellcheck disable=SC2064
+    trap "rm -rf '$tmpdir'" EXIT
+
+    local i
+    for i in "${!SVC_URLS[@]}"; do
+        check_reachability "${SVC_URLS[$i]}" "${tmpdir}/${i}" &
+    done
+    wait
+
+    for i in "${!SVC_URLS[@]}"; do
+        if [[ -f "${tmpdir}/${i}" ]]; then
+            SVC_STATUSES[$i]=$(cat "${tmpdir}/${i}")
+        else
+            SVC_STATUSES[$i]="DOWN"
+        fi
+    done
+}
+
+# --- Output -------------------------------------------------------------------
+
+print_table() {
+    local max_name=7 max_url=3  # minimum: "Service" / "URL"
+    local i
+    for i in "${!SVC_NAMES[@]}"; do
+        [[ ${#SVC_NAMES[$i]} -gt $max_name ]] && max_name=${#SVC_NAMES[$i]}
+        [[ ${#SVC_URLS[$i]} -gt $max_url ]] && max_url=${#SVC_URLS[$i]}
+    done
+
+    local total_width=$((max_name + max_url + 14))
+    local sep
+    sep=$(printf '%*s' "$total_width" '' | tr ' ' '─')
+
+    echo
+    echo -e "${BOLD}Lab Environment: ${ENV_NAME}${RESET}"
+    echo "$sep"
+    printf "%-${max_name}s  %-${max_url}s  %s\n" "Service" "URL" "Status"
+    echo "$sep"
+
+    for i in "${!SVC_NAMES[@]}"; do
+        local status="${SVC_STATUSES[$i]:-—}"
+        local status_colored
+        case "$status" in
+            UP)   status_colored="${GREEN}UP${RESET}" ;;
+            DOWN) status_colored="${RED}DOWN${RESET}" ;;
+            *)    status_colored="$status" ;;
+        esac
+        printf "%-${max_name}s  %-${max_url}s  " "${SVC_NAMES[$i]}" "${SVC_URLS[$i]}"
+        echo -e "$status_colored"
+    done
+
+    echo "$sep"
+    echo
+}
+
+print_json() {
+    local last=$((${#SVC_NAMES[@]} - 1))
+    echo "["
+    local i
+    for i in "${!SVC_NAMES[@]}"; do
+        local comma=","
+        [[ $i -eq $last ]] && comma=""
+        local status_json
+        if [[ -n "${SVC_STATUSES[$i]:-}" ]]; then
+            status_json="\"${SVC_STATUSES[$i]}\""
+        else
+            status_json="null"
+        fi
+        printf '  {"service": "%s", "url": "%s", "status": %s}%s\n' \
+            "${SVC_NAMES[$i]}" "${SVC_URLS[$i]}" "$status_json" "$comma"
+    done
+    echo "]"
+}
+
+# --- Argument parsing ---------------------------------------------------------
+
+parse_args() {
+    while [[ $# -gt 0 ]]; do
+        case "$1" in
+            -e|--env)
+                [[ $# -lt 2 ]] && { error "--env requires a value"; exit 1; }
+                ENV_NAME="$2"; shift
+                ;;
+            --no-check)    NO_CHECK=true ;;
+            --json)        JSON_OUTPUT=true ;;
+            -v|--verbose)  VERBOSE=true ;;
+            -q|--quiet)    QUIET=true ;;
+            -h|--help)     usage; exit 0 ;;
+            *)
+                error "Unknown option: $1"
+                echo "Run '${SCRIPT_NAME} --help' for usage."
+                exit 1
+                ;;
+        esac
+        shift
+    done
+}
+
+# --- Main ---------------------------------------------------------------------
+
+main() {
+    setup_colors
+    parse_args "$@"
+    detect_env
+
+    # Collect services from all config files
+    collect_proxmox
+    collect_lab_vms
+    collect_haproxy
+    collect_awx
+    collect_observability
+
+    if [[ ${#SVC_NAMES[@]} -eq 0 ]]; then
+        warn "No services found in envs/${ENV_NAME}/"
+        exit 0
+    fi
+
+    # Run reachability checks (parallel)
+    if [[ "$NO_CHECK" != true ]]; then
+        info "Checking ${#SVC_NAMES[@]} services..."
+        run_checks
+    fi
+
+    # Output
+    if [[ "$JSON_OUTPUT" == true ]]; then
+        print_json
+    else
+        print_table
+    fi
+}
+
+main "$@"

bin/manage-dashboards

@@ -13,7 +13,7 @@ set -euo pipefail
 readonly VERSION="0.1.0"
 readonly SCRIPT_NAME="manage-dashboards"
 readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-readonly DASHBOARDS_DIR="${SCRIPT_DIR}/observability-dashboards"
+readonly DASHBOARDS_DIR="${SCRIPT_DIR}/data/observability-dashboards"
 readonly MANIFEST_FILE="${DASHBOARDS_DIR}/dashboard.yaml"
 
 # ---------------------------------------------------------------------------

envs/README.md

@@ -0,0 +1,82 @@
+# Environment Configurations
+
+Per-environment connection configs, secrets, and deployment settings. Each
+subdirectory represents a physical host (or host type) with its own network
+topology, storage backend, and resource sizing.
+
+## Directory layout
+
+```
+envs/
+├── beelink/
+│   ├── env                    # Secrets: PROXMOX_TOKEN_SECRET, etc. (gitignored)
+│   ├── proxmox.yaml           # Proxmox connection (gitignored)
+│   ├── proxmox.yaml.example   # Connection template (LAN, vmbr0, local-lvm)
+│   ├── lab-cluster.yaml       # 3-node cluster (4C/4G/50G)
+│   ├── awx.yaml               # AWX deployment settings
+│   ├── haproxy.yaml           # HAProxy deployment settings
+│   └── observability.yaml     # Monitoring stack settings
+└── hetzner/
+    ├── env                    # Secrets (gitignored)
+    ├── proxmox.yaml           # Proxmox connection (gitignored)
+    ├── proxmox.yaml.example   # Connection template (WireGuard, vmbr1, local-zfs)
+    ├── lab-cluster.yaml       # 3-node cluster (8C/16G/100G)
+    ├── lab-production.yaml    # OC + 3-node cluster
+    ├── awx.yaml               # AWX deployment settings
+    ├── haproxy.yaml           # HAProxy deployment settings
+    ├── observability.yaml     # Monitoring stack settings
+    └── setup/                 # Host provisioning (Proxmox install guide + script)
+        ├── README.md
+        ├── hetzner-setup.md
+        ├── hetzner-lab-guide.md
+        └── proxmox-setup
+```
+
+## How to use
+
+### 1. Source the environment
+
+Each environment has its own `env` file with the same variable names.
+Source the one for the host you're targeting:
+
+```bash
+source envs/beelink/env    # or: source envs/hetzner/env
+```
+
+### 2. Deploy VMs
+
+```bash
+bin/incusos-proxmox --proxmox envs/hetzner/proxmox.yaml \
+                    envs/hetzner/lab-production.yaml
+```
+
+### 3. Deploy services with --config
+
+```bash
+bin/deploy-awx --config envs/hetzner/awx.yaml --deploy
+bin/deploy-haproxy --config envs/hetzner/haproxy.yaml --deploy
+bin/deploy-observability --config envs/hetzner/observability.yaml --deploy
+```
+
+## Adding a new environment
+
+1. Create a new subdirectory: `envs/myhost/`
+2. Copy `proxmox.yaml.example` from an existing env and edit
+3. Create an `env` file with your secrets (same variable names)
+4. Create lab YAML files sized for your hardware
+5. Create `awx.yaml`, `haproxy.yaml`, `observability.yaml` with your IPs
+
+## Key differences between environments
+
+| Setting | Beelink | Hetzner |
+|---------|---------|---------|
+| Host | 192.168.1.10 (LAN) | 10.10.0.1 (WireGuard) |
+| Method | ssh | api |
+| Bridge | vmbr0 | vmbr1 (private) |
+| Storage | local-lvm | local-zfs |
+| VLAN | 69 | *(none)* |
+| Gateway | 192.168.100.1 | 10.10.0.1 |
+| VM IPs | 192.168.102.x/22 | 10.10.0.x/24 |
+| VM cores | 4 | 8 |
+| VM memory | 4-8 GiB | 16 GiB |
+| VM disk | 50 GiB | 100 GiB |

envs/beelink/awx.yaml

@@ -0,0 +1,26 @@
+# awx.yaml - AWX deployment config for Beelink lab
+#
+# Deploys a Debian 12 VM with K3s + AWX Operator on oc-node-02.
+# VM uses the flat VLAN 69 network for direct access.
+#
+# Architecture:
+#   awx  (192.168.102.161)  - K3s + AWX Operator + AWX instance
+#   AWX exposed on NodePort 30080 (HTTP)
+#
+# Usage:
+#   source envs/beelink/env
+#   bin/deploy-awx --config envs/beelink/awx.yaml --deploy
+
+awx:
+  vm_name: awx
+  target_remote: oc-node-02
+  ip: 192.168.102.161/22
+  gateway: 192.168.100.1
+  dns: 192.168.100.1
+  cpu: 4
+  memory: 8GiB
+  disk: 40GiB
+  git_repo: ssh://git@192.168.1.200:2222/maarten/incus-contrib.git
+  git_branch: master
+  aether_url: https://192.168.102.160:8443
+  aether_cluster_id: 52

envs/beelink/haproxy.yaml

@@ -0,0 +1,26 @@
+# haproxy.yaml - HAProxy load balancing config for Beelink lab
+#
+# Deploys 2 HAProxy containers on the OVN overlay (net-prod, 10.10.10.0/24)
+# with a VIP on the UPLINK external range (192.168.103.200).
+#
+# Architecture:
+#   ffsdn-haproxy-52-01  (10.10.10.50)  - HAProxy instance 1 on net-prod
+#   ffsdn-haproxy-52-02  (10.10.10.51)  - HAProxy instance 2 on net-prod
+#   nginx-lb-01/02/03    (10.10.10.60-62) - nginx test backends on net-prod
+#   VIP:                  192.168.103.200:80  - OVN LB for external access
+#
+# Usage:
+#   source envs/beelink/env
+#   bin/deploy-haproxy --config envs/beelink/haproxy.yaml --deploy
+
+haproxy:
+  cluster_remote: oc-node-01
+  cluster_id: 52
+  ovn_network: net-prod
+  vip: 192.168.103.200
+  haproxy_01_ip: 10.10.10.50
+  haproxy_02_ip: 10.10.10.51
+  backend_ips: 10.10.10.60,10.10.10.61,10.10.10.62
+  service_name: web-test
+  image_version: "1.0.0"
+  aether_url: https://192.168.102.160:8443

envs/beelink/lab-cluster.yaml

@@ -4,8 +4,8 @@
 # Deploys 3 standalone Incus nodes that can be clustered after deployment.
 #
 # Usage:
-#   incusos-proxmox --proxmox incusos/targets/beelink/proxmox.yaml \
-#                   incusos/targets/beelink/lab-cluster.yaml
+#   bin/incusos-proxmox --proxmox envs/beelink/proxmox.yaml \
+#                       envs/beelink/lab-cluster.yaml
 #
 # Connection settings from proxmox.yaml in this directory.
 

envs/beelink/observability.yaml

@@ -0,0 +1,25 @@
+# observability.yaml - Monitoring stack config for Beelink lab
+#
+# Deploys Prometheus, Grafana, Loki, and Promtail in a monitoring container,
+# plus node-exporter containers on each cluster node.
+#
+# Architecture:
+#   monitoring      (10.10.10.70)     - Prometheus + Grafana + Loki
+#   node-exp-01/02/03                 - node-exporter on each cluster node
+#   Forward VIP:    192.168.103.201   - OVN LB for external access
+#
+# Usage:
+#   source envs/beelink/env
+#   bin/deploy-observability --config envs/beelink/observability.yaml --deploy
+
+observability:
+  cluster_remote: oc-node-01
+  ovn_network: net-prod
+  monitoring_ip: 10.10.10.70
+  monitoring_target: oc-node-02
+  forward_vip: 192.168.103.201
+  incus_nodes: 192.168.102.140,192.168.102.141,192.168.102.142
+  node_exp_targets: oc-node-01,oc-node-02,oc-node-03
+  node_exp_ips: 10.10.10.71,10.10.10.72,10.10.10.73
+  haproxy_ips: 10.10.10.50,10.10.10.51
+  haproxy_cluster_id: 52

envs/hetzner/awx.yaml

@@ -0,0 +1,25 @@
+# awx.yaml - AWX deployment config for Hetzner lab
+#
+# Deploys a Debian 12 VM with K3s + AWX Operator on hz-node-03.
+# VM uses macvlan parent=mgmt for direct access on the 10.10.0.0/24 network.
+#
+# Architecture:
+#   awx  (10.10.0.122)  - K3s + AWX Operator + AWX instance
+#   AWX exposed on NodePort 30080 (HTTP)
+#
+# Usage:
+#   source envs/hetzner/env
+#   bin/deploy-awx --config envs/hetzner/awx.yaml --deploy
+
+awx:
+  vm_name: awx
+  target_remote: hz-cluster
+  target_node: hz-node-03
+  ip: 10.10.0.122/24
+  gateway: 10.10.0.1
+  dns: 10.10.0.1
+  cpu: 4
+  memory: 8GiB
+  disk: 40GiB
+  aether_url: https://10.10.0.120:8443
+  aether_cluster_id: 1

envs/hetzner/haproxy.yaml

@@ -0,0 +1,26 @@
+# haproxy.yaml - HAProxy load balancing config for Hetzner lab
+#
+# Deploys 2 HAProxy containers on the OVN overlay (net-prod, 10.10.10.0/24)
+# with a VIP on the UPLINK external range (10.10.0.200-210).
+#
+# Architecture:
+#   ffsdn-haproxy-1-01   (10.10.10.50)  - HAProxy instance 1 on net-prod
+#   ffsdn-haproxy-1-02   (10.10.10.51)  - HAProxy instance 2 on net-prod
+#   nginx-lb-01/02/03    (10.10.10.60-62) - nginx test backends on net-prod
+#   VIP:                  10.10.0.200:80  - OVN external IP for external access
+#
+# Usage:
+#   source envs/hetzner/env
+#   bin/deploy-haproxy --config envs/hetzner/haproxy.yaml --deploy
+
+haproxy:
+  cluster_remote: hz-cluster
+  cluster_id: 1
+  ovn_network: net-prod
+  vip: 10.10.0.200
+  haproxy_01_ip: 10.10.10.50
+  haproxy_02_ip: 10.10.10.51
+  backend_ips: 10.10.10.60,10.10.10.61,10.10.10.62
+  service_name: web-test
+  image_version: "1.0.0"
+  aether_url: https://10.10.0.120:8443

envs/hetzner/lab-cluster.yaml

@@ -7,8 +7,8 @@
 # VMs get static IPs in the 10.10.0.101-103 range.
 #
 # Usage:
-#   incusos-proxmox --proxmox incusos/targets/hetzner/proxmox.yaml \
-#                   incusos/targets/hetzner/lab-cluster.yaml
+#   bin/incusos-proxmox --proxmox envs/hetzner/proxmox.yaml \
+#                       envs/hetzner/lab-cluster.yaml
 #
 # Connection settings from proxmox.yaml in this directory.
 

envs/hetzner/lab-production.yaml

@@ -1,6 +1,6 @@
 # lab-production.yaml - Production-quality lab for Hetzner (OC + 3-node cluster)
 #
-# Deploys 1 Operations Center instance and 3 Incus nodes for manual clustering.
+# Deploys 1 Operations Center instance and 3 Incus nodes for OC-managed clustering.
 # Sized for a Hetzner dedicated server with generous resources.
 #
 # Architecture:
@@ -12,11 +12,11 @@
 # Network: 10.10.0.0/24 on vmbr1 (private bridge, NAT to internet).
 # OVN overlay (once configured): 10.10.10.0/24 internal, 10.10.0.200+ external.
 #
-# RAM budget: 56 GiB of 256 GiB (22% utilization -- plenty of room for workloads)
+# RAM budget: ~248 GiB of 256 GiB (8 GiB OC + 3×80 GiB nodes; ~8 GiB reserved for Proxmox host)
 #
 # Usage:
-#   incusos-proxmox --proxmox incusos/targets/hetzner/proxmox.yaml \
-#                   incusos/targets/hetzner/lab-production.yaml
+#   bin/incusos-proxmox --proxmox envs/hetzner/proxmox.yaml \
+#                       envs/hetzner/lab-production.yaml
 #
 # After deployment, follow notes/production-lab-guide.md for:
 #   - Cluster formation (section 4)
@@ -26,8 +26,8 @@
 # Connection settings from proxmox.yaml in this directory.
 
 defaults:
-  cores: 8
-  memory: 16384
+  cores: 16
+  memory: 81920            # ~80 GiB per node (3 nodes × 80 = 240 GiB + 8 for OC = 248 GiB total)
   disk: 100
   start_vmid: 910
 
@@ -36,12 +36,12 @@ vms:
     app: operations-center
     apply_defaults: true
     cores: 4
-    memory: 8192
+    memory: 8192            # OC needs very little RAM
     ip: 10.10.0.110/24
 
   - name: hz-node-01
     app: incus
-    apply_defaults: true          # init node: needs storage pool + network
+    apply_defaults: false         # OC manages cluster formation + storage/network setup
     disk: 150                     # extra space for OVN control plane container
     ip: 10.10.0.111/24
 

envs/hetzner/observability.yaml

@@ -0,0 +1,25 @@
+# observability.yaml - Monitoring stack config for Hetzner lab
+#
+# Deploys Prometheus, Grafana, Loki, and Promtail in a monitoring container,
+# plus node-exporter containers on each cluster node.
+#
+# Architecture:
+#   monitoring      (10.10.10.70)     - Prometheus + Grafana + Loki
+#   node-exp-01/02/03                 - node-exporter on each cluster node
+#   Forward VIP:    10.10.0.201       - OVN LB for external access
+#
+# Usage:
+#   source envs/hetzner/env
+#   bin/deploy-observability --config envs/hetzner/observability.yaml --deploy
+
+observability:
+  cluster_remote: hz-cluster
+  ovn_network: net-prod
+  monitoring_ip: 10.10.10.70
+  monitoring_target: hz-node-02
+  forward_vip: 10.10.0.201
+  incus_nodes: 10.10.0.111,10.10.0.112,10.10.0.113
+  node_exp_targets: hz-node-01,hz-node-02,hz-node-03
+  node_exp_ips: 10.10.10.71,10.10.10.72,10.10.10.73
+  haproxy_ips: 10.10.10.50,10.10.10.51
+  haproxy_cluster_id: 1

envs/hetzner/proxmox.yaml

@@ -0,0 +1,12 @@
+host: 10.10.0.1
+method: api
+api_token_id: automation@pve!deploy
+api_token_secret: f7ed66bc-b1c9-4696-b4a9-c64f158c2a51
+node: aether-lab
+storage: local-zfs
+iso_storage: local
+bridge: vmbr1
+gateway: 10.10.0.1
+dns: 10.10.0.1                 # Proxmox host serves DNS for VMs (dnsmasq on vmbr1)
+pool: IncusLab
+ssh_user: root

envs/hetzner/proxmox.yaml.example

@@ -9,11 +9,12 @@
 #
 # Storage: local-zfs (ZFS pool on dedicated disks).
 #
-# See hetzner/hetzner-setup.md for the full setup guide.
+# See envs/hetzner/setup/hetzner-setup.md for the full setup guide.
 
 host: 10.10.0.1               # Proxmox host IP (via WireGuard, on vmbr1)
 method: api                    # api recommended (WireGuard may add SSH latency)
 api_token_id: automation@pve!deploy
+api_token_secret: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx  # token secret (overrides env PROXMOX_TOKEN_SECRET)
 node: pve                      # Proxmox node name (adjust if different)
 storage: local-zfs             # VM disk storage (ZFS pool)
 iso_storage: local             # ISO/image storage

envs/hetzner/setup/README.md

@@ -51,12 +51,12 @@ and API tokens for `incusos-proxmox`.
 ### 4. Deploy IncusOS VMs
 
 ```bash
-cp incusos/targets/hetzner/proxmox.yaml.example incusos/targets/hetzner/proxmox.yaml
+cp envs/hetzner/proxmox.yaml.example envs/hetzner/proxmox.yaml
 # Fill in the API token from step 3
 
 export PROXMOX_TOKEN_SECRET="<token>"
-incusos-proxmox --proxmox incusos/targets/hetzner/proxmox.yaml \
-                incusos/targets/hetzner/lab-cluster.yaml
+bin/incusos-proxmox --proxmox envs/hetzner/proxmox.yaml \
+                    envs/hetzner/lab-cluster.yaml
 ```
 
 ## What's in this directory
@@ -92,5 +92,5 @@ Internet
 
 ## Related files
 
-- [`incusos/targets/hetzner/`](../incusos/targets/hetzner/) -- Proxmox connection configs and lab definitions
-- [`incusos/targets/README.md`](../incusos/targets/README.md) -- Multi-target concept explanation
+- [`envs/hetzner/`](../) -- Proxmox connection configs and lab definitions
+- [`envs/README.md`](../../README.md) -- Multi-target concept explanation

envs/hetzner/setup/hetzner-lab-guide.md

@@ -0,0 +1,907 @@
+# Hetzner IncusOS Lab — End-to-End Guide
+
+Full deployment guide for a production-quality IncusOS lab on a Hetzner
+dedicated server: 1 Operations Center + 3-node Incus cluster with OVN
+networking, HAProxy load balancing, AWX automation, and observability.
+
+**Estimated time**: 3-4 hours for a clean run.
+
+---
+
+## Overview
+
+```
+Hetzner dedicated (256 GB RAM, 32+ cores)
+└─ Proxmox VE 9
+   ├─ hz-oc       (VMID 910) — Operations Center 0.4.x
+   ├─ hz-node-01  (VMID 911) — Incus cluster init + OVN control plane
+   ├─ hz-node-02  (VMID 912) — Incus cluster member
+   └─ hz-node-03  (VMID 913) — Incus cluster member
+      └─ (OVN overlay 10.10.10.0/24, VIP 10.10.0.200+)
+         ├─ ffsdn-haproxy-52-01/02  — HAProxy HA pair
+         ├─ nginx-lb-01/02/03       — Test backends
+         ├─ monitoring              — Prometheus + Grafana
+         ├─ node-exp-01/02/03       — Node exporters
+         └─ awx                     — AWX Operator on K3s
+```
+
+**Network topology:**
+- vmbr0: Public interface (SSH + WireGuard only)
+- vmbr1: Private bridge 10.10.0.0/24 (VMs + NAT)
+- wg0: WireGuard tunnel 10.10.99.0/24 (workstation access)
+- OVN overlay: 10.10.10.0/24 (containers on net-prod)
+- OVN external IPs: 10.10.0.200+ (VIPs exposed on vmbr1)
+
+---
+
+## Prerequisites
+
+- Hetzner dedicated server with Proxmox VE 9 installed and configured
+  (see [hetzner-setup.md](hetzner-setup.md) + `proxmox-setup`)
+- WireGuard tunnel active (10.10.99.x → 10.10.0.x accessible)
+- This repository cloned locally
+- `env` file populated (see below)
+- `envs/hetzner/proxmox.yaml` filled in
+- Incus client installed on workstation
+
+### env file
+
+```bash
+# incusos-contrib/env
+export PROXMOX_TOKEN_SECRET="<token from proxmox-setup>"
+export PROXMOX_ROOT_PASSWORD="<proxmox root password>"
+export AETHER_ADMIN_PASSWORD="admin1234"
+```
+
+### proxmox.yaml
+
+```bash
+cp envs/hetzner/proxmox.yaml.example \
+   envs/hetzner/proxmox.yaml
+# Fill in: host, node name, token_id, storage
+```
+
+---
+
+## 1. Deploy IncusOS VMs
+
+```bash
+source env
+
+bin/incusos-proxmox \
+    --proxmox envs/hetzner/proxmox.yaml \
+    envs/hetzner/lab-production.yaml
+```
+
+**What gets deployed:**
+
+| VMID | Name      | IP            | RAM   | Cores | Disk |
+|------|-----------|---------------|-------|-------|------|
+| 910  | hz-oc     | 10.10.0.110   | 8 GB  | 4     | 100G |
+| 911  | hz-node-01| 10.10.0.111   | 80 GB | 16    | 150G |
+| 912  | hz-node-02| 10.10.0.112   | 80 GB | 16    | 100G |
+| 913  | hz-node-03| 10.10.0.113   | 80 GB | 16    | 100G |
+
+**Boot takes 3-5 minutes** per VM (sysext download from internet). Do not
+interrupt. Monitor progress:
+
+```bash
+# Screenshot VM console (requires PROXMOX_ROOT_PASSWORD in env)
+source env
+bin/helpers/proxmox-screenshot 910   # hz-oc
+bin/helpers/proxmox-screenshot 911   # hz-node-01
+```
+
+Wait until all VMs are reachable:
+
+```bash
+for ip in 10.10.0.110 10.10.0.111 10.10.0.112 10.10.0.113; do
+    echo -n "$ip: "
+    ping -c1 -W2 $ip &>/dev/null && echo "UP" || echo "DOWN"
+done
+```
+
+---
+
+## 2. Add Incus remotes
+
+Add the cluster remote pointing at hz-node-01 (the init node):
+
+```bash
+incus remote add hz-cluster https://10.10.0.111:8443 \
+    --accept-certificate --auth-type tls
+```
+
+> **Note**: The `incus remote add` command prompts for a trust token.
+> Obtain the token from OC after cluster formation (section 3), or
+> if using a pre-formed cluster, generate one with:
+> `incus config trust add --name workstation`
+
+---
+
+## 3. Cluster formation via Operations Center
+
+Open the Operations Center UI:
+
+```
+https://10.10.0.110:8443
+Username: admin
+Password: admin   (first login, then set a new password)
+```
+
+### 3.1 Add Incus nodes
+
+In OC: **Servers → Add Server** for each node:
+
+| Field   | hz-node-01       | hz-node-02       | hz-node-03       |
+|---------|------------------|------------------|------------------|
+| Address | 10.10.0.111:8443 | 10.10.0.112:8443 | 10.10.0.113:8443 |
+| Trust   | via token        | via token        | via token        |
+
+Each node's trust token is displayed on the IncusOS console on first boot.
+
+### 3.2 Form the cluster
+
+In OC: **Clusters → Create Cluster**
+
+- Name: `hz-cluster`
+- Init node: `hz-node-01`
+- Members: `hz-node-02`, `hz-node-03`
+- App seed: `{}` (empty JSON for manual post-formation setup)
+
+OC forms the cluster automatically. Wait for all 3 nodes to show **Ready**.
+
+Cluster ID is assigned by OC — note it (e.g. `52`). You'll need it for
+HAProxy and AWX configuration.
+
+### 3.3 Verify cluster
+
+```bash
+incus cluster list hz-cluster:
+# Should show 3 members: hz-node-01, hz-node-02, hz-node-03
+```
+
+---
+
+## 4. OVN networking
+
+### 4.1 Create the OVN uplink network
+
+The UPLINK network uses the IncusOS management NIC (`mgmt`) — not
+`eth0` or `ens18`. This is specific to IncusOS.
+
+```bash
+# Create UPLINK on each node
+incus network create hz-cluster:UPLINK --type=physical \
+    --target hz-node-01 \
+    config parent=mgmt
+
+incus network create hz-cluster:UPLINK --type=physical \
+    --target hz-node-02 \
+    config parent=mgmt
+
+incus network create hz-cluster:UPLINK --type=physical \
+    --target hz-node-03 \
+    config parent=mgmt
+
+# Finalize UPLINK on cluster level
+incus network create hz-cluster:UPLINK \
+    config ipv4.address=none \
+    config ipv6.address=none
+```
+
+### 4.2 Deploy OVN central
+
+OVN central runs as a privileged container on hz-node-01:
+
+```bash
+incus launch images:debian/12 hz-cluster:ovn-central \
+    --vm=false \
+    --target hz-node-01 \
+    -c security.privileged=true \
+    -d root,size=20GiB
+
+# Wait for agent
+sleep 15
+
+# Install ovn-central
+incus exec hz-cluster:ovn-central -- bash -c "
+    apt-get update -qq
+    apt-get install -y -qq ovn-central
+    systemctl enable ovn-northd ovn-ovsdb-server-nb ovn-ovsdb-server-sb
+    systemctl start ovn-northd ovn-ovsdb-server-nb ovn-ovsdb-server-sb
+"
+```
+
+### 4.3 Expose OVN NB/SB ports to cluster
+
+Create proxy devices so cluster nodes can reach OVN:
+
+```bash
+# Get ovn-central IP
+OVN_IP=$(incus list hz-cluster:ovn-central --format csv -c 4 | head -1 | awk '{print $1}')
+
+incus config device add hz-cluster:ovn-central ovn-nb proxy \
+    listen="tcp:${OVN_IP}:6641" connect="tcp:127.0.0.1:6641" \
+    bind=host
+incus config device add hz-cluster:ovn-central ovn-sb proxy \
+    listen="tcp:${OVN_IP}:6642" connect="tcp:127.0.0.1:6642" \
+    bind=host
+```
+
+### 4.4 Create OVN overlay network
+
+```bash
+# Use hz-node-01 IP as OVN NB/SB server
+NODE01_IP="10.10.0.111"
+
+incus network create hz-cluster:net-prod \
+    --type=ovn \
+    config network=UPLINK \
+    config ipv4.address=10.10.10.1/24 \
+    config ipv4.nat=true \
+    config ipv6.address=none \
+    config ovn.northbound_connection="tcp:${NODE01_IP}:6641" \
+    config ovn.southbound_connection="tcp:${NODE01_IP}:6642"
+```
+
+Wait for the network to reach **Created** state:
+
+```bash
+incus network info hz-cluster:net-prod
+```
+
+### 4.5 Configure external IP range (OVN VIPs)
+
+```bash
+# Reserve 10.10.0.200-210 for OVN external IPs (VIPs)
+# These are allocated on the Proxmox host's vmbr1 bridge
+# No additional configuration needed — OVN will ARP for these IPs
+```
+
+Verify:
+
+```bash
+incus network list hz-cluster:
+# net-prod should show type: ovn, state: Created
+```
+
+---
+
+## 5. Deploy Aether
+
+### 5.1 Deploy the Aether VM
+
+Use the macvlan method to give Aether a direct vmbr1 IP:
+
+```bash
+incus launch images:ubuntu/22.04 hz-cluster:aether \
+    --vm \
+    --target hz-node-01 \
+    -d root,size=40GiB \
+    -c limits.cpu=4 \
+    -c limits.memory=8GiB \
+    -d eth0,type=nic,nictype=macvlan,parent=mgmt,ipv4.address=10.10.0.120
+```
+
+> **Alternative**: use Aether golden image if already built on the cluster.
+> Check `incus image list hz-cluster: | grep aether`.
+
+Wait for the VM, then SSH in:
+
+```bash
+ssh ubuntu@10.10.0.120
+```
+
+Follow the [Aether installation guide](../notes/aether-guide.md) for the
+full Aether setup. Aether is accessible at `https://10.10.0.120:8443`.
+
+### 5.2 Configure Aether cluster
+
+In Aether UI (`https://10.10.0.120:8443`):
+
+1. **Settings → Clusters → Add Cluster**
+   - Name: hz-cluster
+   - Endpoint: `https://10.10.0.111:8443`
+   - Authenticate (generates a trust token, add it to the cluster)
+
+2. **Settings → Operations Centers → Add OC**
+   - URL: `https://10.10.0.110:8443`
+
+3. Verify hz-cluster instances sync in Aether dashboard.
+
+Aether default credentials: `admin` / `admin` (change on first login).
+
+For Aether password, set in `env`:
+
+```bash
+export AETHER_ADMIN_PASSWORD="admin1234"
+```
+
+---
+
+## 6. HAProxy load balancing
+
+### 6.1 Create config file
+
+Save as `envs/hetzner/hz-haproxy.yaml` (or `/tmp/hz-deploy.yaml`):
+
+```yaml
+haproxy:
+  cluster_remote: hz-cluster
+  cluster_id: 52            # from OC cluster listing
+  ovn_network: net-prod
+  vip: 10.10.0.200
+  haproxy_01_ip: 10.10.10.50
+  haproxy_02_ip: 10.10.10.51
+  backend_ips: 10.10.10.60,10.10.10.61,10.10.10.62
+  service_name: web-test
+  image_version: "1.0.0"
+  aether_url: https://10.10.0.120:8443
+```
+
+> **Cluster ID**: Find it via OC API or in Aether UI under Clusters.
+> It's the numeric ID assigned when the cluster was created.
+
+### 6.2 Build HAProxy image
+
+Before deploying infrastructure, build and push the HAProxy image:
+
+In Aether UI → **HAProxy → Images → Build**:
+- Version: `1.0.0`
+- Cluster: `hz-cluster`
+- Click **Build** and wait 5-10 minutes
+
+After build, push to hz-cluster:
+- Click **Push** → select `hz-cluster`
+- Click **Set Current**
+
+Alternatively, use the script (requires the UI steps above for first build):
+
+```bash
+source env
+bin/deploy-haproxy -c /tmp/hz-deploy.yaml --deploy --skip-image
+```
+
+### 6.3 Full deploy
+
+```bash
+source env
+bin/deploy-haproxy -c /tmp/hz-deploy.yaml --deploy
+```
+
+This:
+1. Launches 3 nginx backend containers (`nginx-lb-01/02/03`) on `net-prod`
+2. Deploys HAProxy infrastructure via Aether (2 containers + OVN VIP)
+3. Creates the `web-test` service (HTTP, round-robin, health checks)
+
+On subsequent runs (if backends already exist):
+
+```bash
+bin/deploy-haproxy -c /tmp/hz-deploy.yaml --deploy \
+    --skip-backends --skip-image
+```
+
+### 6.4 Verify
+
+```bash
+curl http://10.10.0.200/
+# Should show "Backend: nginx-lb-0x"
+
+# Test distribution
+for i in $(seq 1 6); do curl -s http://10.10.0.200/ | grep 'Backend:'; done
+```
+
+---
+
+## 7. AWX automation
+
+### 7.1 Create config file
+
+Save as `envs/hetzner/awx.yaml`:
+
+```yaml
+awx:
+  vm_name: awx
+  target_remote: hz-cluster
+  target_node: hz-node-03    # specific node within cluster
+  ip: 10.10.0.122/24
+  gateway: 10.10.0.1
+  dns: 10.10.0.1
+  cpu: 4
+  memory: 8GiB
+  disk: 40GiB
+  aether_url: https://10.10.0.120:8443
+  aether_cluster_id: 52
+```
+
+> **target_node vs target_remote**: `target_remote` is the Incus remote
+> name (e.g. `hz-cluster`), while `target_node` is the specific cluster
+> member to deploy on (e.g. `hz-node-03`). Both are required for
+> cluster deployments.
+
+### 7.2 Deploy
+
+```bash
+source env
+bin/deploy-awx -c envs/hetzner/awx.yaml --deploy
+```
+
+This deploys a Debian 12 VM with K3s + AWX Operator. Takes ~20 minutes.
+AWX will be at `http://10.10.0.122:30080`.
+
+On partial failure (e.g. network issue mid-deploy), resume:
+
+```bash
+bin/deploy-awx -c envs/hetzner/awx.yaml --deploy --resume
+# Skips VM creation and K3s install phases, jumps to AWX Operator phase
+```
+
+If you get `permission denied` on temp files from a previous aborted run:
+
+```bash
+incus exec hz-cluster:awx -- rm -f \
+    /tmp/awx-operator-kustomization.yaml \
+    /tmp/awx-kustomization.yaml \
+    /tmp/awx-instance.yaml
+```
+
+### 7.3 Configure AWX
+
+> **Wait for the awx-task pod before running --configure.** The AWX web
+> deployment becomes ready first (~3 min), but `--configure` needs the
+> task pod. Check it:
+> ```bash
+> incus exec hz-cluster:awx -- kubectl -n awx wait \
+>     --for=condition=Ready pod -l app.kubernetes.io/name=awx-task \
+>     --timeout=300s
+> ```
+
+```bash
+source env
+bin/deploy-awx -c envs/hetzner/awx.yaml --configure
+```
+
+This creates:
+- Project: `incus-contrib` (linked to your git repo)
+- Inventory: `incus-lab`
+- Templates: `post-deploy` (ID 9), `decommission` (ID 10), and lifecycle templates
+
+Get AWX admin password:
+
+```bash
+bin/deploy-awx -c /tmp/hz-awx.yaml --status
+```
+
+### 7.4 Join Aether
+
+Generate an AWX API token for Aether:
+
+```bash
+source env
+bin/deploy-awx -c envs/hetzner/awx.yaml --join-aether
+# Prints the PAT token and template IDs
+```
+
+In Aether UI → **Ansible Automation** → **+ Add Endpoint**:
+- Name: `lab-awx`
+- URL: `http://10.10.0.122:30080`
+- Token: (paste token from `--join-aether` output)
+- Verify SSL: unchecked (HTTP endpoint)
+
+Click **Test Connection** (should show `Connected! AWX version: 24.x.x`) then **Save Endpoint**.
+
+### 7.5 Bind AWX to cluster
+
+**Note**: In Aether ≤ v6.4.202, the `PUT /api/clusters/{id}/awx-config`
+endpoint returns `400 "Invalid cluster ID"` due to a bug.
+
+**Workaround** (direct DB update):
+
+```bash
+# First confirm the AWX endpoint ID (assigned by the database):
+incus exec hz-cluster:aether -- bash -c \
+    "su - postgres -c \"psql -d incusovnsdnc -c 'SELECT id, name, awx_url FROM awx_endpoints;'\""
+
+# Then apply the binding (adjust awx_endpoint_id if yours differs from 2):
+incus exec hz-cluster:aether -- bash -c "su - postgres -c \"psql -d incusovnsdnc -c \\\"
+    UPDATE incus_ovn_clusters
+    SET awx_endpoint_id=2,
+        awx_post_deploy_template_id=9,
+        awx_decommission_template_id=10,
+        awx_job_timeout_seconds=600
+    WHERE cluster_id=52;
+\\\"\""
+```
+
+> **Note**: The database is PostgreSQL (`incusovnsdnc`), accessed as the
+> `postgres` system user. The endpoint ID is auto-assigned — always check
+> it before applying the fix. Template IDs 9 and 10 are set by `--configure`.
+
+> **Check future Aether versions**: Run `curl -sk https://10.10.0.120:8443/api/swagger.yaml | grep awx-config`
+> to see if the endpoint has been fixed. If it returns a valid schema,
+> use the API instead of the DB workaround.
+
+Verify in Aether UI: **Clusters → hz-cluster → Settings** should show
+the AWX endpoint, post-deploy, and decommission templates.
+
+---
+
+## 8. Observability stack
+
+### 8.1 Deploy
+
+```bash
+source env
+bin/deploy-observability \
+    --remote hz-cluster \
+    --monitoring-target hz-node-01 \
+    --monitoring-ip 10.10.10.70 \
+    --forward-vip 10.10.0.201 \
+    --deploy
+```
+
+> **Note**: Do NOT pass `--oc-server` for the Hetzner setup. The
+> Operations Center (hz-oc) runs OC, not standard Incus — its
+> `/1.0/instances` endpoint returns 404 and will fail the node
+> discovery phase.
+
+> **Grafana install may fail if net-prod containers lack internet.**
+> OVN SNAT (outbound from 10.10.10.0/24 via 10.10.0.200) works briefly
+> after container launch but can become unreliable. If phase 4 fails:
+>
+> ```bash
+> # Download Grafana locally and push it in
+> curl -L -o /tmp/grafana.deb https://dl.grafana.com/oss/release/grafana_12.4.0_amd64.deb
+> incus file push /tmp/grafana.deb hz-cluster:monitoring/tmp/grafana.deb
+> incus exec hz-cluster:monitoring -- bash -c "
+>     export DEBIAN_FRONTEND=noninteractive
+>     dpkg -i /tmp/grafana.deb; apt-get -f install -y
+>     mkdir -p /etc/grafana/provisioning/datasources
+>     cat > /etc/grafana/provisioning/datasources/observability.yaml << 'EOF'
+> apiVersion: 1
+> datasources:
+>   - name: Prometheus
+>     type: prometheus
+>     uid: prometheus
+>     access: proxy
+>     url: http://localhost:9090
+>     isDefault: true
+>   - name: Loki
+>     type: loki
+>     uid: loki
+>     access: proxy
+>     url: http://localhost:3100
+> EOF
+>     systemctl daemon-reload
+>     systemctl enable grafana-server
+>     systemctl start grafana-server
+> "
+> ```
+>
+> Then continue with `--resume --deploy` (skips phases 1-5, runs ACLs,
+> network forward, and dashboards).
+
+> **Node-exporter containers are skipped by `--resume`.** If internet
+> was unavailable during phase 5, deploy them manually:
+>
+> ```bash
+> # Download node_exporter binary locally
+> curl -sL -o /tmp/ne.tar.gz \
+>     https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz
+> tar xzf /tmp/ne.tar.gz -C /tmp node_exporter-1.8.2.linux-amd64/node_exporter
+>
+> for i in 1 2 3; do
+>   name="node-exp-0${i}"
+>   ip="10.10.10.7${i}"
+>   node="hz-node-0${i}"
+>
+>   incus launch images:alpine/3.20 hz-cluster:${name} \
+>     -n net-prod --target ${node} \
+>     -c limits.memory=128MiB -c security.privileged=true
+>   incus config device set hz-cluster:${name} eth0 ipv4.address=${ip}
+>   sleep 5
+>
+>   for dev in proc sys root; do
+>     src="/$( [[ $dev == root ]] && echo '' || echo $dev)"; src="${src:-/}"
+>     incus config device add hz-cluster:${name} host-${dev} disk \
+>       source=/${dev} path=/host/${dev} readonly=true 2>/dev/null || \
+>     incus config device add hz-cluster:${name} host-${dev} disk \
+>       source=/ path=/host/root readonly=true 2>/dev/null || true
+>   done
+>
+>   incus file push /tmp/node_exporter-1.8.2.linux-amd64/node_exporter \
+>     hz-cluster:${name}/usr/bin/node_exporter
+>
+>   incus exec hz-cluster:${name} -- sh -c "
+>     chmod +x /usr/bin/node_exporter
+>     cat > /etc/init.d/node-exporter << 'INIT'
+> #!/sbin/openrc-run
+> description=\"Node Exporter\"
+> command=/usr/bin/node_exporter
+> command_args=\"--path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/host/root\"
+> command_background=true
+> pidfile=/run/node-exporter.pid
+> INIT
+>     chmod +x /etc/init.d/node-exporter
+>     rc-update add node-exporter default
+>     rc-service node-exporter start
+>
+>     cat > /etc/network/interfaces << NETCFG
+> auto lo
+> iface lo inet loopback
+> auto eth0
+> iface eth0 inet static
+>     address ${ip}/24
+>     gateway 10.10.10.1
+> NETCFG
+>     rc-service networking restart
+>   "
+>
+>   incus config device set hz-cluster:${name} eth0 \
+>     security.acls=monitoring-allow \
+>     security.acls.default.ingress.action=allow \
+>     security.acls.default.egress.action=allow
+> done
+> ```
+
+On partial failure (phases 6-9 only), resume:
+
+```bash
+source env
+bin/deploy-observability \
+    --remote hz-cluster \
+    --monitoring-target hz-node-01 \
+    --monitoring-ip 10.10.10.70 \
+    --forward-vip 10.10.0.201 \
+    --resume --deploy
+```
+
+After deploying, fix the HAProxy prometheus exporter (default stats page
+doesn't expose `/metrics`):
+
+```bash
+for ct in ffsdn-haproxy-52-01 ffsdn-haproxy-52-02; do
+  incus exec hz-cluster:${ct} -- bash -c "
+    sed -i 's/    stats enable/    http-request use-service prometheus-exporter if { path \/metrics }\n    stats enable/' /etc/haproxy/haproxy.cfg
+    systemctl reload haproxy
+  "
+done
+```
+
+### 8.2 Access
+
+| Service    | URL                          | Credentials |
+|------------|------------------------------|-------------|
+| Grafana    | http://10.10.0.201:3000      | admin/admin |
+| Prometheus | http://10.10.0.201:9090      | —           |
+
+9 dashboards are pre-loaded: cluster overview, node metrics, HAProxy traffic,
+storage, networking, capacity planning, and log explorer.
+
+Verify all Prometheus targets are up:
+
+```bash
+curl -s http://10.10.0.201:9090/api/v1/targets | python3 -c "
+import sys,json
+for t in json.load(sys.stdin)['data']['activeTargets']:
+    print(t['labels'].get('job','?'), t['health'], t['scrapeUrl'][:50])
+"
+# Expected: 2 haproxy, 3 incus, 3 node-exporter, 1 prometheus — all 'up'
+```
+
+---
+
+## 9. WireGuard access (macOS)
+
+If you use the macOS WireGuard app from the App Store:
+
+- The app uses Network Extension (visible as `utun5` in `ifconfig`)
+- `wg show` always shows empty — this is normal with the App Store app
+- The app allows only **one active tunnel at a time**
+
+For a second simultaneous tunnel:
+
+```bash
+# Install wg-quick for macOS
+brew install wireguard-tools
+
+# Start the second tunnel alongside the App Store one
+sudo wg-quick up ~/.config/wireguard/hetzner.conf
+```
+
+If the server added a new WireGuard peer (new workstation/client):
+
+```bash
+# On the Hetzner Proxmox host (via the existing tunnel or public IP)
+ssh hetzner-lab "wg set wg0 peer <public-key> allowed-ips 10.10.99.x/32"
+ssh hetzner-lab "wg-quick save wg0"
+```
+
+---
+
+## 10. Incus remote management
+
+### Add remotes
+
+```bash
+# OC (Operations Center — not standard Incus)
+# Note: Must be added manually — incus remote add loops prompting for token
+# OC does not support the standard trust token flow.
+# Edit ~/.config/incus/config.yml directly:
+```
+
+Add to `~/.config/incus/config.yml`:
+
+```yaml
+remotes:
+  oc-server:
+    addr: https://10.10.0.110:8443
+    auth_type: tls
+    protocol: incus
+    public: false
+  hz-cluster:
+    addr: https://10.10.0.111:8443
+    auth_type: tls
+    protocol: incus
+    public: false
+```
+
+Copy the OC certificate (accept it with curl first):
+
+```bash
+# Accept and save OC cert
+curl -sk https://10.10.0.110:8443 > /dev/null
+# The cert ends up in your browser cache — use openssl to fetch it
+openssl s_client -connect 10.10.0.110:8443 </dev/null 2>/dev/null \
+    | openssl x509 > ~/.config/incus/servercerts/oc-server.crt
+```
+
+> **Warning**: `oc-server` is OC, not Incus. `incus list oc-server:` will
+> fail. OC does not expose the standard `/1.0/instances` endpoint.
+
+### After redeploy (remote cert changed)
+
+```bash
+# Remove old remote and re-add
+incus remote remove hz-cluster
+incus remote add hz-cluster https://10.10.0.111:8443 \
+    --accept-certificate --auth-type tls
+```
+
+---
+
+## 11. Quick reference
+
+### Access URLs
+
+| Service         | URL                              | Credentials         |
+|-----------------|----------------------------------|---------------------|
+| Operations Center | https://10.10.0.110:8443       | admin / (set on first login) |
+| Aether          | https://10.10.0.120:8443         | admin / admin1234   |
+| AWX             | http://10.10.0.122:30080         | admin / (from deploy-awx --status) |
+| Grafana         | http://10.10.0.201:3000          | admin / admin       |
+| Prometheus      | http://10.10.0.201:9090          | —                   |
+| HAProxy VIP     | http://10.10.0.200/              | —                   |
+
+### IP allocation
+
+| Range                 | Purpose                      |
+|-----------------------|------------------------------|
+| 10.10.0.1             | Proxmox host (vmbr1 gateway) |
+| 10.10.0.110-113       | IncusOS VMs (OC + nodes)     |
+| 10.10.0.120           | Aether VM (macvlan)          |
+| 10.10.0.122           | AWX VM (macvlan)             |
+| 10.10.0.200+          | OVN external IPs (VIPs)      |
+| 10.10.10.0/24         | OVN overlay (containers)     |
+| 10.10.10.50-51        | HAProxy containers           |
+| 10.10.10.60-62        | nginx test backends          |
+| 10.10.10.70           | monitoring container         |
+| 10.10.10.71-73        | node-exporter containers     |
+| 10.10.99.0/24         | WireGuard tunnel             |
+| 10.10.99.1            | WireGuard server             |
+| 10.10.99.2            | Your workstation             |
+
+### Cluster IDs
+
+- hz-cluster (OC): ID 52 (assigned by OC at cluster creation — check
+  your OC UI if different)
+
+### Useful incus commands
+
+```bash
+# List all instances in cluster
+incus list hz-cluster:
+
+# List cluster members
+incus cluster list hz-cluster:
+
+# List networks
+incus network list hz-cluster:
+
+# Exec into a container
+incus exec hz-cluster:monitoring -- bash
+
+# Start/stop a container
+incus start hz-cluster:monitoring
+incus stop hz-cluster:monitoring --force
+
+# Check container network
+incus exec hz-cluster:nginx-lb-01 -- ip a
+```
+
+### Teardown
+
+```bash
+# Force destroy all VMs (no nice cleanup needed for test lab)
+source env
+for vmid in 910 911 912 913; do
+    bin/helpers/proxmox-api POST \
+        "/nodes/aether-lab/qemu/${vmid}/status/stop" \
+        '{"forceStop":1}' 2>/dev/null || true
+    sleep 3
+    bin/helpers/proxmox-api DELETE \
+        "/nodes/aether-lab/qemu/${vmid}?purge=1&destroy-unreferenced-disks=1"
+done
+
+# Remove stale remotes
+incus remote remove hz-cluster 2>/dev/null || true
+incus remote remove oc-server 2>/dev/null || true
+```
+
+---
+
+## Known issues and workarounds
+
+### OC remote add loops (incus remote add oc-server fails)
+
+`incus remote add` enters a trust token prompt loop for OC because OC
+does not implement the standard Incus trust token flow.
+
+**Workaround**: Add the remote manually to `~/.config/incus/config.yml`
+and save the TLS certificate separately.
+
+### IncusOS management NIC name is `mgmt`, not `eth0`
+
+All IncusOS instances use `mgmt` as the management NIC name. Use
+`parent=mgmt` in UPLINK network config, not `parent=eth0` or `parent=ens18`.
+
+### Aether `PUT /api/clusters/{id}/awx-config` returns 400
+
+Bug in Aether ≤ v6.4.202. Use the direct DB workaround (section 7.5).
+
+### deploy-observability: don't use --oc-server with OC
+
+The `--oc-server` flag tries to list instances via the standard Incus
+API (`/1.0/instances`), which OC does not implement. Omit the flag.
+
+### OVN SNAT internet access from net-prod containers is unreliable
+
+Containers on `net-prod` (10.10.10.0/24) use SNAT via 10.10.0.200 to
+reach the internet. This works briefly after container launch (via Proxmox
+NAT masquerade) but can become unreliable as ARP for the VIP IP becomes
+stale. Connections return `Connection refused` from Fastly CDN.
+
+**Workaround**: Download packages locally and push them in with
+`incus file push` instead of relying on apt/apk inside net-prod containers.
+
+Containers on macvlan mgmt (like AWX, Aether) have reliable internet via
+Proxmox masquerade and are unaffected.
+
+### deploy-awx temp file permission errors on resume
+
+Temp files created as root inside the AWX VM persist between runs.
+Clean up before resuming:
+
+```bash
+incus exec hz-cluster:awx -- rm -f \
+    /tmp/awx-operator-kustomization.yaml \
+    /tmp/awx-kustomization.yaml \
+    /tmp/awx-instance.yaml
+```
+
+### aether_url in config files must use key-specific sed
+
+Config parsing uses `sed 's/^ *aether_url: *//'` (key-specific match).
+A greedy `sed 's/.*: *//'` would strip the `:8443` port. This is
+already fixed in all scripts in this repo.

envs/hetzner/setup/hetzner-setup.md

@@ -295,7 +295,7 @@ Each step prompts for confirmation. Use `--yes` to skip prompts, or
 `--skip-repos`, `--skip-storage`, etc. to skip individual steps.
 
 At the end, the script prints:
-- A ready-to-use `proxmox.yaml` config for `incusos/targets/hetzner/`
+- A ready-to-use `proxmox.yaml` config for `envs/hetzner/`
 - An `env` file snippet with the API token
 - A WireGuard client config to save on your workstation
 
@@ -330,15 +330,15 @@ Host hetzner-lab-wg
 
 ```bash
 # Copy and fill in the connection config
-cp incusos/targets/hetzner/proxmox.yaml.example incusos/targets/hetzner/proxmox.yaml
+cp envs/hetzner/proxmox.yaml.example envs/hetzner/proxmox.yaml
 
 # Set the API token
 export PROXMOX_TOKEN_SECRET="<token from proxmox-setup output>"
 
 # Dry run -- verify correct bridge, IPs, storage
-incusos-proxmox --dry-run \
-    --proxmox incusos/targets/hetzner/proxmox.yaml \
-    incusos/targets/hetzner/lab-cluster.yaml
+bin/incusos-proxmox --dry-run \
+    --proxmox envs/hetzner/proxmox.yaml \
+    envs/hetzner/lab-cluster.yaml
 ```
 
 Confirm the output shows: `vmbr1` bridge, `10.10.0.x` IPs, `local-zfs`

envs/hetzner/setup/proxmox-setup

@@ -34,7 +34,6 @@ WG_PORT="51820"
 SKIP_REPOS=false
 SKIP_WIREGUARD=false
 SKIP_FIREWALL=false
-SKIP_STORAGE=false
 
 # Derived (computed after argument parsing)
 SUBNET_BASE=""          # e.g. 10.10.0
@@ -72,12 +71,7 @@ DETECT_VMBR1_IP=""              # e.g. "10.10.0.1/24" or empty
 DETECT_IP_FORWARD=""            # "1" or "0"
 DETECT_NAT_ACTIVE=""            # "yes" or empty
 
-# Storage
-STATE_STORAGE="done"            # done | available | needed
 DETECT_ZPOOLS=""
-DETECT_PVE_STORAGE=""
-DETECT_AVAILABLE_DISKS=""
-DETECT_AVAILABLE_DISK_COUNT="0"
 
 # WireGuard
 STATE_WIREGUARD="needed"        # done | needed
@@ -91,11 +85,16 @@ STATE_FIREWALL="needed"         # done | needed
 DETECT_NFT_RULES=""             # "yes" if nftables rules loaded with our pattern
 DETECT_NFT_FILE=""              # "yes" if config file exists
 
+# DNS (dnsmasq on vmbr1 -- required for IncusOS boot)
+STATE_DNSMASQ="needed"          # done | needed
+DETECT_DNSMASQ_ACTIVE=""        # "yes" if dnsmasq is installed and listening on vmbr1
+
 # API token
 STATE_API="needed"              # done | partial | needed
 DETECT_API_ROLE=""              # "yes" or empty
 DETECT_API_POOL=""              # "yes" or empty
 DETECT_API_TOKEN=""             # "yes" or empty
+DETECT_API_ACLS=""              # "yes" if /vms ACL is set
 
 # ---------------------------------------------------------------------------
 # Color and formatting
@@ -126,9 +125,9 @@ detail()  { [[ "$VERBOSE" != true ]] && return; echo -e "${DIM}          $*${RES
 dry_run_guard() {
     if [[ "$DRY_RUN" == true ]]; then
         info "(dry-run) $*"
-        return 1
-    fi
         return 0
+    fi
+    return 1
 }
 
 # Status indicator for the dashboard
@@ -206,7 +205,6 @@ ${BOLD}DESCRIPTION${RESET}
 
     - No-subscription repositories and system update
     - Private bridge (vmbr1) with NAT masquerading
-    - ZFS storage pool on extra disks
     - WireGuard tunnel for secure remote access
     - nftables firewall lockdown
     - API token for incusos-proxmox automation
@@ -220,7 +218,6 @@ ${BOLD}OPTIONS${RESET}
     ${BOLD}--wg-subnet CIDR${RESET}      WireGuard subnet (default: ${WG_SUBNET})
     ${BOLD}--wg-port PORT${RESET}        WireGuard listen port (default: ${WG_PORT})
     ${BOLD}--skip-repos${RESET}          Skip repository changes
-    ${BOLD}--skip-storage${RESET}        Skip storage pool creation
     ${BOLD}--skip-wireguard${RESET}      Skip WireGuard setup
     ${BOLD}--skip-firewall${RESET}       Skip firewall lockdown
     ${BOLD}-n, --dry-run${RESET}         Preview actions without executing
@@ -250,7 +247,7 @@ ${BOLD}PREREQUISITES${RESET}
 
 ${BOLD}SEE ALSO${RESET}
     hetzner/hetzner-setup.md    Manual installation guide
-    incusos/targets/hetzner/    Target config files
+    envs/hetzner/               Target config files
 EOF
     exit 0
 }
@@ -282,10 +279,6 @@ parse_args() {
                 SKIP_REPOS=true
                 shift
                 ;;
-            --skip-storage)
-                SKIP_STORAGE=true
-                shift
-                ;;
             --skip-wireguard)
                 SKIP_WIREGUARD=true
                 shift
@@ -370,31 +363,79 @@ detect_all() {
     fi
     success "SSH connection to ${SSH_HOST}"
 
-    # Gather everything in a single SSH call for speed
+    # Gather everything in a single SSH call for speed.
+    # Uses a quoted heredoc (<<'DETECT') so nothing is expanded locally --
+    # all $, ", etc. are passed literally to the remote shell.
     local detection_blob
-    detection_blob=$(remote_output "
+    detection_blob=$(ssh -o ConnectTimeout=10 -o BatchMode=yes "$SSH_HOST" bash <<'DETECT' 2>/dev/null || true
 echo '---HOSTNAME---'
 hostname
 echo '---KERNEL---'
 uname -r
 echo '---PVE_VERSION---'
-        pvesh get /version --output-format json 2>/dev/null | python3 -c 'import sys,json; print(json.load(sys.stdin)[\"version\"])' 2>/dev/null || echo 'unknown'
+pvesh get /version --output-format json 2>/dev/null \
+    | python3 -c 'import sys,json; print(json.load(sys.stdin)["version"])' 2>/dev/null \
+    || echo 'unknown'
 echo '---PUBLIC_IP---'
-        hostname -I | awk '{print \$1}'
+hostname -I | awk '{print $1}'
 echo '---BLOCK_DEVICES---'
 lsblk -d -o NAME,SIZE,MODEL,ROTA,TYPE 2>/dev/null
 echo '---ZPOOLS---'
 zpool list -H 2>/dev/null || true
-        echo '---PVE_STORAGE---'
-        pvesm status 2>/dev/null || true
 echo '---ENTERPRISE_REPOS---'
-        grep -rl '^deb.*enterprise' /etc/apt/sources.list.d/*.list 2>/dev/null || echo 'none'
+# Check both .list (PVE 8) and .sources/deb822 (PVE 9) formats
+enterprise_active='no'
+# Old .list format: uncommented deb lines with enterprise
+if grep -rl '^deb.*enterprise' /etc/apt/sources.list.d/*.list 2>/dev/null | head -1 | grep -q .; then
+    enterprise_active='yes'
+fi
+# New .sources/deb822 format: check for enterprise files that are NOT disabled
+for f in /etc/apt/sources.list.d/*enterprise*.sources /etc/apt/sources.list.d/*enterprise*.list; do
+    [ -f "$f" ] || continue
+    # deb822: Enabled: false means disabled
+    if grep -qi '^Enabled:.*false' "$f" 2>/dev/null; then
+        continue
+    fi
+    # deb822: if it has enterprise in URIs and no Enabled: false, it's active
+    if grep -qi 'enterprise' "$f" 2>/dev/null; then
+        enterprise_active='yes'
+    fi
+done
+echo "$enterprise_active"
 echo '---NOSUB_REPO---'
-        test -f /etc/apt/sources.list.d/pve-no-subscription.list && echo 'yes' || echo 'no'
+# Check both .list and .sources formats for no-subscription repo
+nosub='no'
+if grep -rq 'pve-no-subscription' /etc/apt/sources.list.d/ 2>/dev/null; then
+    nosub='yes'
+fi
+echo "$nosub"
 echo '---NAG_PATCHED---'
-        grep -q 'void({' /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js 2>/dev/null && echo 'yes' || echo 'no'
+# Detect whether the subscription nag popup has been removed.
+# Methods vary by tool and PVE version:
+#   1. community-scripts post-install: installs a dpkg hook that patches
+#      proxmoxlib.js after every update, changing "!== 'active'" to
+#      "== 'NoMoreNagging'" on data.status lines. Leaves behind:
+#      - /etc/apt/apt.conf.d/no-nag-script (dpkg hook)
+#      - /usr/local/bin/pve-remove-nag.sh (the patcher)
+#   2. pve-nag-buster: similar dpkg hook approach
+#   3. Manual sed: replaces Ext.Msg.show with void({
+nag_patched='no'
+# Check for persistent dpkg hook (community-scripts / pve-nag-buster)
+if [ -f /etc/apt/apt.conf.d/no-nag-script ] || [ -f /usr/local/bin/pve-remove-nag.sh ]; then
+    nag_patched='yes'
+# Check for pve-nag-buster package hook
+elif ls /etc/dpkg/dpkg.cfg.d/*nag* /etc/apt/apt.conf.d/*nag-buster* 2>/dev/null | grep -q .; then
+    nag_patched='yes'
+# Check JS for void({ patch (manual sed method)
+elif grep -q 'void({' /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js 2>/dev/null; then
+    nag_patched='yes'
+# Check if nag text was removed entirely
+elif ! grep -q "No valid subscription" /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js 2>/dev/null; then
+    nag_patched='yes'
+fi
+echo "$nag_patched"
 echo '---UPGRADABLE---'
-        apt list --upgradable 2>/dev/null | grep -c upgradable || echo '0'
+apt list --upgradable 2>/dev/null | grep -c upgradable || true
 echo '---VMBR1---'
 ip -br addr show vmbr1 2>/dev/null || echo 'none'
 echo '---IP_FORWARD---'
@@ -404,31 +445,47 @@ detect_all() {
 echo '---WG_INSTALLED---'
 command -v wg >/dev/null 2>&1 && echo 'yes' || echo 'no'
 echo '---WG_ACTIVE---'
-        wg show wg0 2>/dev/null | head -1 || echo 'none'
+wg_out=$(wg show wg0 2>/dev/null | head -1) || true
+if [ -n "$wg_out" ]; then echo "$wg_out"; else echo 'none'; fi
 echo '---WG_PEERS---'
-        wg show wg0 peers 2>/dev/null | wc -l || echo '0'
+wg_peers=$(wg show wg0 peers 2>/dev/null | wc -l) || true
+echo "${wg_peers:-0}"
 echo '---WG_HANDSHAKE---'
-        wg show wg0 latest-handshakes 2>/dev/null | head -1 || echo 'none'
+wg_hs=$(wg show wg0 latest-handshakes 2>/dev/null | head -1) || true
+if [ -n "$wg_hs" ]; then echo "$wg_hs"; else echo 'none'; fi
 echo '---NFT_FILE---'
 test -f /etc/nftables-hetzner.conf && echo 'yes' || echo 'no'
 echo '---NFT_RULES---'
 nft list ruleset 2>/dev/null | grep -q 'nft-drop' && echo 'yes' || echo 'no'
+echo '---DNSMASQ---'
+# dnsmasq installed and listening on port 53 on vmbr1?
+dnsmasq_ok='no'
+if command -v dnsmasq >/dev/null 2>&1 && systemctl is-active dnsmasq >/dev/null 2>&1; then
+    # Check it's listening on the bridge IP
+    if ss -ulnp 2>/dev/null | grep ':53 ' | grep -q dnsmasq; then
+        dnsmasq_ok='yes'
+    fi
+fi
+echo "$dnsmasq_ok"
 echo '---API_ROLE---'
-        pveum role list --output-format json 2>/dev/null | python3 -c 'import sys,json; any(r[\"roleid\"]==\"IncusOSDeployer\" for r in json.load(sys.stdin)) and print(\"yes\")' 2>/dev/null || echo 'no'
+pveum role list --output-format json 2>/dev/null \
+    | python3 -c 'import sys,json; print("yes" if any(r["roleid"]=="IncusOSDeployer" for r in json.load(sys.stdin)) else "no")' 2>/dev/null \
+    || echo 'no'
 echo '---API_POOL---'
-        pveum pool list --output-format json 2>/dev/null | python3 -c 'import sys,json; any(p[\"poolid\"]==\"IncusLab\" for p in json.load(sys.stdin)) and print(\"yes\")' 2>/dev/null || echo 'no'
+pveum pool list --output-format json 2>/dev/null \
+    | python3 -c 'import sys,json; print("yes" if any(p["poolid"]=="IncusLab" for p in json.load(sys.stdin)) else "no")' 2>/dev/null \
+    || echo 'no'
 echo '---API_TOKEN---'
-        pveum user token list automation@pve --output-format json 2>/dev/null | python3 -c 'import sys,json; any(t[\"tokenid\"]==\"deploy\" for t in json.load(sys.stdin)) and print(\"yes\")' 2>/dev/null || echo 'no'
-        echo '---AVAILABLE_DISKS---'
-        used_disks=\$(zpool status 2>/dev/null | grep -oP '/dev/\\S+' | sed 's|/dev/||' || true)
-        lsblk -dno NAME,SIZE,TYPE | while read name size type; do
-            [ \"\$type\" != 'disk' ] && continue
-            echo \"\$used_disks\" | grep -qw \"\$name\" && continue
-            lsblk -no MOUNTPOINT /dev/\$name 2>/dev/null | grep -q '/' && continue
-            echo \"\$name \$size\"
-        done
+pveum user token list automation@pve --output-format json 2>/dev/null \
+    | python3 -c 'import sys,json; print("yes" if any(t["tokenid"]=="deploy" for t in json.load(sys.stdin)) else "no")' 2>/dev/null \
+    || echo 'no'
+echo '---API_ACLS---'
+pveum acl list --output-format json 2>/dev/null \
+    | python3 -c 'import sys,json; acls=json.load(sys.stdin); paths={a.get("path") for a in acls if "automation@pve" in a.get("ugid","")}; ok=("/vms" in paths and "/sdn" in paths); print("yes" if ok else "no")' 2>/dev/null \
+    || echo 'no'
 echo '---END---'
-    " || true)
+DETECT
+)
 
     # Parse the blob into variables
     local section=""
@@ -451,7 +508,7 @@ ${line}"
         fi
     done <<< "$detection_blob"
     # Process last section
-    [[ -n "$section" ]] && _parse_section "$section" "$section_lines"
+    if [[ -n "$section" ]]; then _parse_section "$section" "$section_lines"; fi
 
     # Determine composite states
     _compute_states
@@ -469,12 +526,11 @@ _parse_section() {
         PUBLIC_IP)      DETECT_PUBLIC_IP="$content" ;;
         BLOCK_DEVICES)  DETECT_BLOCK_DEVICES="$content" ;;
         ZPOOLS)         DETECT_ZPOOLS="$content" ;;
-        PVE_STORAGE)    DETECT_PVE_STORAGE="$content" ;;
         ENTERPRISE_REPOS)
-            if [[ "$content" == "none" ]]; then
-                DETECT_ENTERPRISE_ACTIVE=""
+            if [[ "$content" == "yes" ]]; then
+                DETECT_ENTERPRISE_ACTIVE="yes"
             else
-                DETECT_ENTERPRISE_ACTIVE="$content"
+                DETECT_ENTERPRISE_ACTIVE=""
             fi
             ;;
         NOSUB_REPO)     DETECT_NOSUB_EXISTS="$content" ;;
@@ -489,34 +545,32 @@ _parse_section() {
         NAT_ACTIVE)     DETECT_NAT_ACTIVE="$content" ;;
         WG_INSTALLED)   DETECT_WG_INSTALLED="$content" ;;
         WG_ACTIVE)
-            if [[ "$content" != "none" ]]; then
+            if [[ "$content" != "none" ]] && [[ -n "$content" ]]; then
                 DETECT_WG_ACTIVE="yes"
             fi
             ;;
         WG_PEERS)       DETECT_WG_PEERS="${content//[^0-9]/}" ;;
         WG_HANDSHAKE)
-            if [[ "$content" != "none" ]]; then
+            if [[ "$content" != "none" ]] && [[ -n "$content" ]]; then
                 DETECT_WG_HANDSHAKE="$content"
             fi
             ;;
         NFT_FILE)       DETECT_NFT_FILE="$content" ;;
         NFT_RULES)      DETECT_NFT_RULES="$content" ;;
+        DNSMASQ)
+            if [[ "$content" == "yes" ]]; then DETECT_DNSMASQ_ACTIVE="yes"; fi
+            ;;
         API_ROLE)
-            [[ "$content" == "yes" ]] && DETECT_API_ROLE="yes"
+            if [[ "$content" == "yes" ]]; then DETECT_API_ROLE="yes"; fi
             ;;
         API_POOL)
-            [[ "$content" == "yes" ]] && DETECT_API_POOL="yes"
+            if [[ "$content" == "yes" ]]; then DETECT_API_POOL="yes"; fi
             ;;
         API_TOKEN)
-            [[ "$content" == "yes" ]] && DETECT_API_TOKEN="yes"
+            if [[ "$content" == "yes" ]]; then DETECT_API_TOKEN="yes"; fi
             ;;
-        AVAILABLE_DISKS)
-            DETECT_AVAILABLE_DISKS="$content"
-            if [[ -n "$content" ]]; then
-                DETECT_AVAILABLE_DISK_COUNT=$(echo "$content" | wc -l)
-            else
-                DETECT_AVAILABLE_DISK_COUNT="0"
-            fi
+        API_ACLS)
+            if [[ "$content" == "yes" ]]; then DETECT_API_ACLS="yes"; fi
             ;;
     esac
 }
@@ -549,21 +603,6 @@ _compute_states() {
         STATE_BRIDGE="needed"
     fi
 
-    # Storage: check if any ZFS pool is registered as PVE storage
-    local zfs_in_pve
-    zfs_in_pve=$(echo "$DETECT_PVE_STORAGE" | grep -c "zfspool" || true)
-    if [[ "$zfs_in_pve" -gt 0 ]]; then
-        if [[ "$DETECT_AVAILABLE_DISK_COUNT" -gt 0 ]]; then
-            STATE_STORAGE="available"  # has pool but also unused disks
-        else
-            STATE_STORAGE="done"
-        fi
-    elif [[ "$DETECT_AVAILABLE_DISK_COUNT" -gt 0 ]]; then
-        STATE_STORAGE="needed"
-    else
-        STATE_STORAGE="done"  # no pool, but no disks available either
-    fi
-
     # WireGuard
     if [[ "$DETECT_WG_ACTIVE" == "yes" ]]; then
         STATE_WIREGUARD="done"
@@ -579,13 +618,20 @@ _compute_states() {
     fi
 
     # API token
-    if [[ "$DETECT_API_ROLE" == "yes" ]] && [[ "$DETECT_API_POOL" == "yes" ]] && [[ "$DETECT_API_TOKEN" == "yes" ]]; then
+    if [[ "$DETECT_API_ROLE" == "yes" ]] && [[ "$DETECT_API_POOL" == "yes" ]] && [[ "$DETECT_API_TOKEN" == "yes" ]] && [[ "$DETECT_API_ACLS" == "yes" ]]; then
         STATE_API="done"
     elif [[ "$DETECT_API_ROLE" == "yes" ]] || [[ "$DETECT_API_POOL" == "yes" ]] || [[ "$DETECT_API_TOKEN" == "yes" ]]; then
         STATE_API="partial"
     else
         STATE_API="needed"
     fi
+
+    # dnsmasq (DNS for VMs on vmbr1 -- required for IncusOS boot)
+    if [[ "$DETECT_DNSMASQ_ACTIVE" == "yes" ]]; then
+        STATE_DNSMASQ="done"
+    else
+        STATE_DNSMASQ="needed"
+    fi
 }
 
 # ---------------------------------------------------------------------------
@@ -634,14 +680,12 @@ show_status() {
 
     local effective_repos="$STATE_REPOS"
     local effective_update="$STATE_UPDATE"
-    local effective_storage="$STATE_STORAGE"
     local effective_wg="$STATE_WIREGUARD"
     local effective_fw="$STATE_FIREWALL"
 
-    [[ "$SKIP_REPOS" == true ]]      && effective_repos="skipped" && effective_update="skipped"
-    [[ "$SKIP_STORAGE" == true ]]    && effective_storage="skipped"
-    [[ "$SKIP_WIREGUARD" == true ]]  && effective_wg="skipped"
-    [[ "$SKIP_FIREWALL" == true ]]   && effective_fw="skipped"
+    if [[ "$SKIP_REPOS" == true ]]; then effective_repos="skipped"; effective_update="skipped"; fi
+    if [[ "$SKIP_WIREGUARD" == true ]]; then effective_wg="skipped"; fi
+    if [[ "$SKIP_FIREWALL" == true ]]; then effective_fw="skipped"; fi
 
     printf "  %-20s %b" "Repositories" "$(status_icon "$effective_repos")"
     case "$effective_repos" in
@@ -672,14 +716,6 @@ show_status() {
         needed)  echo "  vmbr1 not yet created" ;;
     esac
 
-    printf "  %-20s %b" "Storage" "$(status_icon "$effective_storage")"
-    case "$effective_storage" in
-        done)      echo "  ZFS pool registered" ;;
-        available) echo "  ZFS pool exists, ${DETECT_AVAILABLE_DISK_COUNT} unused disk(s) available" ;;
-        needed)    echo "  ${DETECT_AVAILABLE_DISK_COUNT} unused disk(s), no pool created" ;;
-        skipped)   echo "" ;;
-    esac
-
     printf "  %-20s %b" "WireGuard" "$(status_icon "$effective_wg")"
     case "$effective_wg" in
         done)    echo "  wg0 active, ${DETECT_WG_PEERS:-0} peer(s)" ;;
@@ -698,9 +734,9 @@ show_status() {
         done)    echo "  nftables rules active" ;;
         needed)
             if [[ "$DETECT_NFT_FILE" == "yes" ]]; then
-                echo "  config file exists but rules not loaded"
+                echo "  config written, apply manually (see firewall step)"
             else
-                echo "  no rules configured"
+                echo "  config not yet written"
             fi
             ;;
         skipped) echo "" ;;
@@ -719,17 +755,23 @@ show_status() {
         needed)  echo "  not configured" ;;
     esac
 
+    printf "  %-20s %b" "DNS (dnsmasq)" "$(status_icon "$STATE_DNSMASQ")"
+    case "$STATE_DNSMASQ" in
+        done)   echo "  dnsmasq active, forwarding DNS on vmbr1" ;;
+        needed) echo "  not installed (required for IncusOS boot)" ;;
+    esac
+
     echo ""
 
     # Count pending items
     local pending=0
-    [[ "$effective_repos" == "needed" || "$effective_repos" == "partial" ]] && pending=$((pending + 1))
-    [[ "$effective_update" == "needed" ]] && pending=$((pending + 1))
-    [[ "$STATE_BRIDGE" == "needed" || "$STATE_BRIDGE" == "partial" ]] && pending=$((pending + 1))
-    [[ "$effective_storage" == "needed" ]] && pending=$((pending + 1))
-    [[ "$effective_wg" == "needed" ]] && pending=$((pending + 1))
-    [[ "$effective_fw" == "needed" ]] && pending=$((pending + 1))
-    [[ "$STATE_API" == "needed" || "$STATE_API" == "partial" ]] && pending=$((pending + 1))
+    if [[ "$effective_repos" == "needed" || "$effective_repos" == "partial" ]]; then pending=$((pending + 1)); fi
+    if [[ "$effective_update" == "needed" ]]; then pending=$((pending + 1)); fi
+    if [[ "$STATE_BRIDGE" == "needed" || "$STATE_BRIDGE" == "partial" ]]; then pending=$((pending + 1)); fi
+    if [[ "$effective_wg" == "needed" ]]; then pending=$((pending + 1)); fi
+    if [[ "$effective_fw" == "needed" ]]; then pending=$((pending + 1)); fi
+    if [[ "$STATE_API" == "needed" || "$STATE_API" == "partial" ]]; then pending=$((pending + 1)); fi
+    if [[ "$STATE_DNSMASQ" == "needed" ]]; then pending=$((pending + 1)); fi
 
     if [[ "$pending" -eq 0 ]]; then
         success "All configured -- nothing to do"
@@ -776,13 +818,30 @@ apply_repos() {
     fi
 
     if [[ -n "$DETECT_ENTERPRISE_ACTIVE" ]]; then
-        remote "sed -i 's/^deb/# deb/' /etc/apt/sources.list.d/pve-enterprise.list 2>/dev/null || true"
-        remote "test -f /etc/apt/sources.list.d/ceph.list && sed -i 's/^deb/# deb/' /etc/apt/sources.list.d/ceph.list || true"
+        # Handle both .list (PVE 8 / bookworm) and .sources (PVE 9 / trixie) formats
+        # .list format: comment out deb lines
+        remote "for f in /etc/apt/sources.list.d/*enterprise*.list; do [ -f \"\$f\" ] && sed -i 's/^deb/# deb/' \"\$f\"; done 2>/dev/null || true"
+        # .sources/deb822 format: set Enabled: false
+        remote "for f in /etc/apt/sources.list.d/*enterprise*.sources; do [ -f \"\$f\" ] && sed -i '/^Enabled:/d' \"\$f\" && echo 'Enabled: false' >> \"\$f\"; done 2>/dev/null || true"
         success "Enterprise repos disabled"
     fi
 
     if [[ "$DETECT_NOSUB_EXISTS" != "yes" ]]; then
+        # Detect PVE version to choose format
+        local pve_major="${DETECT_PVE_VERSION%%.*}"
+        if [[ "${pve_major:-8}" -ge 9 ]]; then
+            # PVE 9+ uses deb822 .sources format
+            remote "cat > /etc/apt/sources.list.d/proxmox.sources <<'REPO'
+Types: deb
+URIs: http://download.proxmox.com/debian/pve
+Suites: trixie
+Components: pve-no-subscription
+Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
+REPO"
+        else
+            # PVE 8 uses traditional .list format
             remote "echo 'deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription' > /etc/apt/sources.list.d/pve-no-subscription.list"
+        fi
         success "No-subscription repo added"
     fi
 
@@ -908,94 +967,46 @@ iface vmbr1 inet static
 }
 
 # ---------------------------------------------------------------------------
-# Apply: storage
+# Apply: dnsmasq (DNS for VMs on vmbr1)
 # ---------------------------------------------------------------------------
 
-apply_storage() {
-    if [[ "$SKIP_STORAGE" == true ]] || [[ "$STATE_STORAGE" == "done" ]]; then
+apply_dnsmasq() {
+    if [[ "$STATE_DNSMASQ" == "done" ]]; then
         return 0
     fi
 
-    step "ZFS storage pool"
+    step "DNS (dnsmasq on vmbr1)"
 
-    if [[ "$STATE_STORAGE" == "available" ]]; then
-        info "ZFS pool already registered with Proxmox"
-        info "${DETECT_AVAILABLE_DISK_COUNT} additional unused disk(s) available:"
-        echo "$DETECT_AVAILABLE_DISKS" | while IFS= read -r line; do
-            [[ -n "$line" ]] && echo "          /dev/$line"
-        done
-        if ! confirm "Create an additional storage pool?"; then
-            return 0
-        fi
-    fi
+    info "Will: install dnsmasq, configure it to forward DNS queries from VMs on vmbr1"
+    info "  Interface: vmbr1 (${SUBNET_GW})"
+    info "  Upstream:  1.1.1.1, 8.8.8.8"
+    info "  Required for IncusOS boot (downloads updates from images.linuxcontainers.org)"
 
-    if [[ "$STATE_STORAGE" == "needed" ]]; then
-        # Check if a zpool named local-zfs already exists but isn't in PVE
-        local existing_pool
-        existing_pool=$(remote_output "zpool list -H local-zfs 2>/dev/null" || true)
-        if [[ -n "$existing_pool" ]]; then
-            info "ZFS pool 'local-zfs' exists but not registered with Proxmox"
-            if confirm "Register local-zfs with Proxmox?"; then
-                if ! dry_run_guard "pvesm add zfspool local-zfs"; then
-                    remote "pvesm add zfspool local-zfs -pool local-zfs"
-                    remote "pvesm set local-zfs -content images,rootdir"
-                    success "local-zfs registered"
-                fi
-            fi
-            return 0
-        fi
-    fi
-
-    if [[ "$DETECT_AVAILABLE_DISK_COUNT" -eq 0 ]]; then
-        info "No unused disks found"
-        return 0
-    fi
-
-    info "Available disks:"
-    echo "$DETECT_AVAILABLE_DISKS" | while IFS= read -r line; do
-        [[ -n "$line" ]] && echo "          /dev/$line"
-    done
-
-    local disk_count="$DETECT_AVAILABLE_DISK_COUNT"
-    local disk_names
-    disk_names=$(echo "$DETECT_AVAILABLE_DISKS" | awk 'NF{print "/dev/"$1}' | tr '\n' ' ')
-
-    local pool_type
-    if [[ "$disk_count" -eq 1 ]]; then
-        pool_type="single"
-        info "Recommendation: single disk (no redundancy)"
-    elif [[ "$disk_count" -eq 2 ]]; then
-        pool_type="mirror"
-        info "Recommendation: mirror (2 disks, full redundancy)"
-    else
-        pool_type="raidz1"
-        info "Recommendation: raidz1 (${disk_count} disks, 1-disk redundancy)"
-    fi
-
-    local pool_name
-    pool_name=$(prompt_value "Pool name" "local-zfs")
-
-    if ! confirm "Create ZFS pool '${pool_name}' (${pool_type}) with: ${disk_names}?"; then
+    if ! confirm "Install and configure dnsmasq?"; then
         info "Skipped"
         return 0
     fi
 
-    if dry_run_guard "zpool create ${pool_name} ${pool_type} ${disk_names}"; then
-        return 0
-    fi
+    dry_run_guard
 
-    local zpool_cmd="zpool create ${pool_name}"
-    case "$pool_type" in
-        mirror) zpool_cmd="${zpool_cmd} mirror ${disk_names}" ;;
-        raidz1) zpool_cmd="${zpool_cmd} raidz1 ${disk_names}" ;;
-        single) zpool_cmd="${zpool_cmd} ${disk_names}" ;;
-    esac
+    remote "apt-get install -y dnsmasq"
 
-    remote "$zpool_cmd"
-    remote "pvesm add zfspool ${pool_name} -pool ${pool_name}"
-    remote "pvesm set ${pool_name} -content images,rootdir"
+    local dnsmasq_conf
+    dnsmasq_conf="# DNS for VMs on vmbr1 (${SUBNET})
+# Generated by proxmox-setup -- do not edit manually
+interface=vmbr1
+bind-interfaces
+no-resolv
+server=1.1.1.1
+server=8.8.8.8
+domain-needed
+bogus-priv"
 
-    success "ZFS pool '${pool_name}' created and registered"
+    remote_write "/etc/dnsmasq.d/vmbr1.conf" "$dnsmasq_conf"
+    remote "systemctl enable dnsmasq && systemctl restart dnsmasq"
+    remote "ss -ulnp 2>/dev/null | grep ':53 ' | grep -q dnsmasq && echo 'OK' || echo 'WARN: dnsmasq not listening on port 53'"
+
+    success "dnsmasq configured on vmbr1"
 }
 
 # ---------------------------------------------------------------------------
@@ -1019,6 +1030,21 @@ apply_wireguard() {
         return 0
     fi
 
+    if [[ "$DRY_RUN" == true ]]; then
+        info "(dry-run) Client config will look like:"
+        echo ""
+        echo "  [Interface]"
+        echo "  PrivateKey = <generated on setup>"
+        echo "  Address = ${WG_CLIENT_IP}/${WG_SUBNET_MASK}"
+        echo ""
+        echo "  [Peer]"
+        echo "  PublicKey = <server public key>"
+        echo "  Endpoint = ${DETECT_PUBLIC_IP:-<server-public-ip>}:${WG_PORT}"
+        echo "  AllowedIPs = 10.10.0.0/16"
+        echo "  PersistentKeepalive = 25"
+        echo ""
+    fi
+
     if dry_run_guard "Install and configure WireGuard (${WG_SERVER_IP}:${WG_PORT})"; then
         return 0
     fi
@@ -1081,6 +1107,22 @@ AllowedIPs = ${WG_CLIENT_IP}/32"
     echo "  PersistentKeepalive = 25"
     echo ""
     warn "Save the client config now -- private key is not stored anywhere else"
+    echo ""
+    info "Verify WireGuard before applying firewall rules"
+    if confirm "Connect your WireGuard client now and verify the tunnel?"; then
+        info "Checking server-side WireGuard status..."
+        local wg_status
+        wg_status=$(remote_output "wg show wg0 2>/dev/null" || true)
+        if echo "$wg_status" | grep -q "latest handshake"; then
+            success "WireGuard is working -- handshake confirmed"
+        elif echo "$wg_status" | grep -q "peer:"; then
+            warn "Peer configured but no handshake yet -- client connected?"
+            info "Try: ping ${WG_SERVER_IP}"
+        else
+            warn "No peers visible -- check wg0 is up on server"
+            remote_output "systemctl status wg-quick@wg0 --no-pager -l" || true
+        fi
+    fi
 }
 
 # ---------------------------------------------------------------------------
@@ -1094,31 +1136,6 @@ apply_firewall() {
 
     step "Firewall lockdown (nftables)"
 
-    if [[ "$DETECT_NFT_FILE" == "yes" ]]; then
-        info "Config file exists but rules not loaded"
-        if confirm "Load existing firewall rules from /etc/nftables-hetzner.conf?"; then
-            if ! dry_run_guard "nft -f /etc/nftables-hetzner.conf"; then
-                remote "nft -f /etc/nftables-hetzner.conf"
-                remote "systemctl enable nftables"
-                success "Firewall rules loaded"
-            fi
-        fi
-        return 0
-    fi
-
-    info "Will restrict public interface to SSH (22) and WireGuard (${WG_PORT}) only"
-    info "Web UI (8006) and VM traffic only via WireGuard"
-    warn "Make sure WireGuard is working before applying firewall rules!"
-
-    if ! confirm "Apply firewall rules?"; then
-        info "Skipped"
-        return 0
-    fi
-
-    if dry_run_guard "Apply nftables rules (SSH + WG only on public interface)"; then
-        return 0
-    fi
-
     local nft_rules="#!/usr/sbin/nft -f
 
 flush ruleset
@@ -1170,12 +1187,26 @@ table inet filter {
     }
 }"
 
+    if [[ "$DETECT_NFT_FILE" != "yes" ]]; then
+        if dry_run_guard "Write /etc/nftables-hetzner.conf on ${SSH_HOST}"; then
+            info "Rules that would be written to /etc/nftables-hetzner.conf:"
+            echo "$nft_rules" | sed 's/^/    /'
+            echo ""
+        else
             remote_write "/etc/nftables-hetzner.conf" "$nft_rules"
-    remote "nft -f /etc/nftables-hetzner.conf"
-    remote "cp /etc/nftables-hetzner.conf /etc/nftables.conf"
-    remote "systemctl enable nftables"
+            success "Config written to /etc/nftables-hetzner.conf"
+        fi
+    else
+        info "Config already at /etc/nftables-hetzner.conf (rules not yet loaded)"
+    fi
 
-    success "Firewall applied (SSH + WireGuard only on public interface)"
+    warn "Firewall rules are NOT applied automatically — misfire locks out SSH on Hetzner permanently"
+    info "Once WireGuard is verified, apply manually:"
+    echo ""
+    echo "  ssh ${SSH_HOST} nft -f /etc/nftables-hetzner.conf"
+    echo "  ssh ${SSH_HOST} systemctl enable nftables"
+    echo "  ssh ${SSH_HOST} nft list ruleset    # verify"
+    echo ""
 }
 
 # ---------------------------------------------------------------------------
@@ -1194,6 +1225,7 @@ apply_api_token() {
     [[ "$DETECT_API_ROLE" == "yes" ]]  || actions="create IncusOSDeployer role"
     [[ "$DETECT_API_POOL" == "yes" ]]  || actions="${actions:+$actions, }create IncusLab pool"
     [[ "$DETECT_API_TOKEN" == "yes" ]] || actions="${actions:+$actions, }create automation@pve!deploy token"
+    [[ "$DETECT_API_ACLS" == "yes" ]]  || actions="${actions:+$actions, }add missing ACLs (/vms, /sdn, storage)"
 
     info "Will: ${actions}"
 
@@ -1210,9 +1242,8 @@ apply_api_token() {
     if [[ "$DETECT_API_ROLE" != "yes" ]]; then
         remote "pveum role add IncusOSDeployer -privs \
             'VM.Allocate VM.Config.CDROM VM.Config.CPU VM.Config.Disk VM.Config.HWType \
-             VM.Config.Memory VM.Config.Network VM.Config.Options VM.PowerMgmt \
-             VM.Monitor VM.Audit VM.Console \
-             Datastore.AllocateSpace Datastore.Audit \
+             VM.Config.Memory VM.Config.Network VM.Config.Options VM.PowerMgmt VM.Audit \
+             Datastore.AllocateSpace Datastore.AllocateTemplate Datastore.Audit \
              Pool.Allocate Pool.Audit \
              SDN.Use Sys.Modify'"
         success "Created IncusOSDeployer role"
@@ -1235,8 +1266,22 @@ apply_api_token() {
         local token_secret
         token_secret=$(echo "$token_output" | python3 -c "import sys,json; print(json.load(sys.stdin)['value'])" 2>/dev/null || echo "$token_output")
 
-        # Assign ACLs
+        success "API token created"
+        echo ""
+        warn "Token secret (save now -- shown only once):"
+        echo ""
+        echo "  ${token_secret}"
+        echo ""
+    fi
+
+    # Assign ACLs (idempotent -- run whenever any ACL is missing).
+    # PVE 9.x requires /sdn for bridge-backed VMs even with SDN.Use in the role.
+    # /vms and /nodes/<node> are required for VM creation and start via API.
+    if [[ "$DETECT_API_ACLS" != "yes" ]]; then
+        remote "pveum acl modify /vms -user automation@pve -role IncusOSDeployer"
+        remote "pveum acl modify /nodes/${DETECT_HOSTNAME} -user automation@pve -role IncusOSDeployer"
         remote "pveum acl modify /pool/IncusLab -user automation@pve -role IncusOSDeployer"
+        remote "pveum acl modify /sdn -user automation@pve -role IncusOSDeployer"
 
         local storage_name
         storage_name=$(remote_output "pvesm status 2>/dev/null | awk '/zfspool/{print \$1}' | head -1" || true)
@@ -1245,12 +1290,7 @@ apply_api_token() {
         fi
         remote "pveum acl modify /storage/local -user automation@pve -role IncusOSDeployer"
 
-        success "API token created"
-        echo ""
-        warn "Token secret (save now -- shown only once):"
-        echo ""
-        echo "  ${token_secret}"
-        echo ""
+        success "ACLs configured"
     fi
 }
 
@@ -1262,7 +1302,7 @@ show_summary() {
     step "Summary"
 
     echo ""
-    echo -e "${BOLD}proxmox.yaml for incusos/targets/hetzner/:${RESET}"
+    echo -e "${BOLD}proxmox.yaml for envs/hetzner/:${RESET}"
     echo ""
     echo "  host: ${SUBNET_GW}"
     echo "  method: api"
@@ -1281,18 +1321,17 @@ show_summary() {
     echo ""
     echo "  # Hetzner"
     echo "  HETZNER_PROXMOX_TOKEN_SECRET=<token from step above>"
-    echo "  HETZNER_PROXMOX_ROOT_PASSWORD=<your root password>"
 
     echo ""
     echo -e "${BOLD}Next steps:${RESET}"
     echo ""
     echo "  1. Save WireGuard client config and connect"
     echo "  2. Copy proxmox.yaml.example and fill in credentials:"
-    echo "     cp incusos/targets/hetzner/proxmox.yaml.example incusos/targets/hetzner/proxmox.yaml"
+    echo "     cp envs/hetzner/proxmox.yaml.example envs/hetzner/proxmox.yaml"
     echo "  3. Test with dry run:"
     echo "     export PROXMOX_TOKEN_SECRET=\"\$HETZNER_PROXMOX_TOKEN_SECRET\""
-    echo "     incusos-proxmox --dry-run --proxmox incusos/targets/hetzner/proxmox.yaml \\"
-    echo "                     incusos/targets/hetzner/lab-cluster.yaml"
+    echo "     bin/incusos-proxmox --dry-run --proxmox envs/hetzner/proxmox.yaml \\"
+    echo "                         envs/hetzner/lab-cluster.yaml"
     echo ""
 }
 
@@ -1332,7 +1371,7 @@ main() {
     apply_repos
     apply_update
     apply_bridge
-    apply_storage
+    apply_dnsmasq
     apply_wireguard
     apply_firewall
     apply_api_token

incusos/targets/README.md

@@ -1,86 +0,0 @@
-# Target Configurations
-
-Target-specific Proxmox connection configs and lab definitions. Each
-subdirectory represents a physical host (or host type) with its own
-network topology, storage backend, and resource sizing.
-
-## How it works
-
-`incusos-proxmox` already supports `--proxmox FILE` to point at any
-connection config. Targets are just an organizational convention --
-no script changes required.
-
-## Directory layout
-
-```
-targets/
-├── beelink/
-│   ├── proxmox.yaml.example     # Connection template (LAN, vmbr0, local-lvm)
-│   └── lab-cluster.yaml          # 3-node cluster sized for beelink (4C/4G/50G)
-└── hetzner/
-    ├── proxmox.yaml.example     # Connection template (WireGuard, vmbr1, local-zfs)
-    ├── lab-cluster.yaml          # 3-node cluster sized for hetzner (8C/16G/100G)
-    └── lab-production.yaml       # OC + 3-node cluster for hetzner
-```
-
-## Usage
-
-### 1. Copy the example and fill in credentials
-
-```bash
-cp incusos/targets/hetzner/proxmox.yaml.example incusos/targets/hetzner/proxmox.yaml
-# Edit with your API token, host IP, etc.
-```
-
-### 2. Deploy with explicit --proxmox
-
-```bash
-incusos-proxmox --proxmox incusos/targets/hetzner/proxmox.yaml \
-                incusos/targets/hetzner/lab-cluster.yaml
-```
-
-### 3. Or symlink for convenience
-
-```bash
-# Point the auto-discovered proxmox.yaml at your active target
-ln -sf targets/hetzner/proxmox.yaml incusos/proxmox.yaml
-
-# Now deploy without --proxmox
-incusos-proxmox incusos/targets/hetzner/lab-cluster.yaml
-```
-
-## Environment variables
-
-`incusos-proxmox` reads `PROXMOX_TOKEN_SECRET` from the environment (or
-the `env` file at the repo root). When working with multiple targets,
-use different variable names per host:
-
-```bash
-# In env file
-PROXMOX_TOKEN_SECRET=beelink-token-here
-HETZNER_PROXMOX_TOKEN_SECRET=hetzner-token-here
-```
-
-Before deploying to a specific target, export the right secret:
-
-```bash
-# Deploy to Hetzner
-export PROXMOX_TOKEN_SECRET="$HETZNER_PROXMOX_TOKEN_SECRET"
-incusos-proxmox --proxmox incusos/targets/hetzner/proxmox.yaml \
-                incusos/targets/hetzner/lab-cluster.yaml
-```
-
-## Key differences between targets
-
-| Setting | Beelink | Hetzner |
-|---------|---------|---------|
-| Host | 192.168.1.10 (LAN) | 10.10.0.1 (WireGuard) |
-| Method | ssh | api |
-| Bridge | vmbr0 | vmbr1 (private) |
-| Storage | local-lvm | local-zfs |
-| VLAN | 69 | *(none)* |
-| Gateway | 192.168.100.1 | 10.10.0.1 |
-| VM IPs | 192.168.102.x/22 | 10.10.0.x/24 |
-| VM cores | 4 | 8 |
-| VM memory | 4-8 GiB | 16 GiB |
-| VM disk | 50 GiB | 100 GiB |

notes/aether-inventory-post-redeploy.md

@@ -0,0 +1,131 @@
+# Aether Inventory After Redeploy (Without DB Backup)
+
+**Date**: 2026-03-04
+**Target**: Hetzner lab, Aether at 10.10.0.120
+**Version**: 6.4.440 (up from 6.4.202 at time of redeploy)
+**Activation code**: 443F4EEC6E088017 (new — DB reinstall generates a new one)
+
+## Summary
+
+Aether was redeployed from a golden image without restoring the previous
+database. This inventory documents what survived (auto-discovered from
+the live cluster) versus what was lost (stored only in the DB).
+
+### What survived (auto-discovered from cluster)
+
+These are things Aether reconstructs by connecting to the Incus cluster
+and querying the API. They require no database state.
+
+| Feature | Status | Detail |
+|---------|--------|--------|
+| Cluster connection | **Present** | hz-cluster (ID 53, was 52 before redeploy) |
+| Cluster members | **Present** | 3/3 nodes online, fully operational |
+| Instance inventory | **Present** | 12 instances, all Running, correct IPs and locations |
+| Instance tags | **Present** | HAProxy tags (ha-group, managed, role) visible |
+| Storage pools | **Present** | local (zfs), 6% used (7GB/112.4GB) |
+| Networking/OVN | **Present** | Visible via Manage INCUS Clusters > Networking |
+| Cluster health | **Present** | Health dashboard shows CPU/mem/load per node |
+
+**Key insight**: Aether's core value — seeing your cluster and instances —
+works immediately after a fresh deploy + cluster add. The Incus API is the
+source of truth for all infrastructure state; Aether caches it but can
+rebuild from scratch.
+
+### What was lost (DB-only state)
+
+These are things Aether stores in its PostgreSQL database that have no
+external source of truth. They must be manually recreated.
+
+| Feature | Status | Detail |
+|---------|--------|--------|
+| Blueprints | **Empty** | "No blueprints created yet" — all designs gone |
+| Deployed blueprints | **Empty** | No deployment history — tracking of which blueprints were deployed where is lost |
+| Operations Center connections | **Empty** | No OC configured — previous OC link gone |
+| Global firewall rules | **Empty** | 0 rules (API confirms: `[]`) |
+| Cluster firewall rules | **Empty** | 0 rules (API confirms: `[]`) |
+| RBAC users/roles | **Default only** | Only the built-in admin account exists |
+| License | **Gone** | New activation code generated — previous license key invalid |
+| Ledger / audit log | **Empty** | No historical log entries |
+| Sync logs | **Empty** | No sync history |
+| AWX job history | **Empty** | "No AWX job history yet" — past job runs not tracked |
+| HAProxy image push state | **Not pushed** | Image v2.64 exists locally but shows "No Image" for hz-cluster — needs re-push |
+| Auto-backup setting | **Off** | `auto_backup_enabled: False` (default) |
+
+### What was re-added manually (post-redeploy)
+
+These items exist in the current database because they were manually
+reconfigured after the redeploy.
+
+| Feature | Status | Detail |
+|---------|--------|--------|
+| AWX endpoint | **Configured** | "lab-awx" at http://10.10.0.122:30080, created 2026-03-03 11:17:36 |
+| AWX cluster config | **Configured** | hz-cluster linked to lab-awx, post-deploy template 9, decommission template 10, 600s timeout |
+| AWX health | **Healthy** | Version 24.6.1, reachable |
+
+## Observations
+
+### Cluster ID changed (52 → 53)
+
+The previous Aether instance had the cluster registered as ID 52 (visible
+in haproxy.yaml and awx.yaml configs). After redeploy, the new database
+assigned it ID 53. This is a synthetic auto-increment ID internal to
+Aether's DB. The cluster itself (certificate, nodes, instances) is
+unchanged — only Aether's internal reference number differs.
+
+**Impact**: Scripts that hardcode `cluster_id: 52` (haproxy.yaml, awx.yaml)
+would need updating if they interact with Aether's API using this ID.
+The deploy scripts use the cluster_id for Aether API calls (HAProxy image
+push, AWX cluster config). If these scripts are re-run against this Aether
+instance, the ID mismatch would cause failures.
+
+### HAProxy image survived, push state didn't
+
+The HAProxy base image v2.64 (287 MB, built 2026-02-21) is stored on
+Aether's local filesystem, not in the DB. It survived the redeploy. But
+the DB record of which clusters it was pushed to was lost. The UI shows
+"No Image" for hz-cluster, meaning the image needs to be re-pushed before
+HAProxy LB management works through Aether.
+
+The HAProxy containers themselves (ffsdn-haproxy-52-01/02) are running
+fine in the cluster — they don't depend on Aether for runtime operation.
+Aether only manages their configuration and deployment lifecycle.
+
+### Settings are all defaults
+
+All settings reverted to defaults. Notable:
+- `log_level: Debug` (default, verbose — consider changing to Info)
+- `refresh_interval_minutes: 5`
+- `auto_backup_enabled: False` — **should enable this time**
+- `manage_instances: True`
+- `manage_vcenters: False`
+- `migration_manager: False`
+
+### What this tells us about Aether's architecture
+
+1. **Cluster state is ephemeral in Aether** — it's a cache of the Incus API.
+   Losing the DB doesn't lose infrastructure visibility. Re-add the cluster
+   and everything reappears within one sync cycle (5 min default).
+
+2. **Operational config is DB-only** — blueprints, ACL rules, RBAC, AWX
+   endpoints, OC connections, and deployment history live exclusively in
+   PostgreSQL. No export/import mechanism was observed.
+
+3. **The license is tied to the DB** — reinstalling generates a new
+   activation code, invalidating any existing license key. This is
+   explicitly noted on the licensing page.
+
+4. **HAProxy images are filesystem-based** — they survive DB loss but
+   lose their cluster push state. The containers in the cluster are
+   independent of Aether once deployed.
+
+5. **AWX integration is lightweight** — just an endpoint URL + template IDs.
+   Quick to re-add manually (as was done). Job history is lost but AWX
+   itself retains its own job history.
+
+## Recovery priority if this happens again
+
+1. Enable `auto_backup_enabled` immediately after setup
+2. Periodically copy `/opt/ffsdn/backups/` off the Aether VM
+3. After restore: re-add cluster, re-add AWX endpoint, re-push HAProxy image
+4. Blueprints and ACL rules would need to be recreated from scratch
+   (consider documenting them externally)