Compare commits
No commits in common. "4bb7a551f216c5a3e32b0f0d341da5fdfb0a739d" and "513d7cd0c6afadda5fe1830a499d46a0f41062a4" have entirely different histories.
4bb7a551f2
...
513d7cd0c6
|
|
@ -1,7 +1,7 @@
|
||||||
# Aether API Payloads Context
|
# Aether API Payloads Context
|
||||||
|
|
||||||
paths:
|
paths:
|
||||||
- bin/helpers/incus-mitm
|
- incus-mitm
|
||||||
- api-interception
|
- api-interception
|
||||||
|
|
||||||
## How Aether Communicates with Incus
|
## How Aether Communicates with Incus
|
||||||
|
|
@ -20,11 +20,11 @@ prefix is `6cfd2a1949a7`.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Live stream with Aether filter
|
# Live stream with Aether filter
|
||||||
bin/helpers/incus-mitm --live --filter aether
|
incusos/helpers/incus-mitm --live --filter aether
|
||||||
|
|
||||||
# Capture to file
|
# Capture to file
|
||||||
bin/helpers/incus-mitm --capture 120
|
incusos/helpers/incus-mitm --capture 120
|
||||||
bin/helpers/incus-mitm --analyze /tmp/incus-events-*.json --filter aether
|
incusos/helpers/incus-mitm --analyze /tmp/incus-events-*.json --filter aether
|
||||||
```
|
```
|
||||||
|
|
||||||
### API Call Patterns
|
### API Call Patterns
|
||||||
|
|
|
||||||
|
|
@ -1,8 +1,8 @@
|
||||||
---
|
---
|
||||||
paths:
|
paths:
|
||||||
- "ansible/**"
|
- "ansible/**"
|
||||||
- "bin/deploy-awx"
|
- "incusos/deploy-awx"
|
||||||
- "bin/data/awx-manifests/*"
|
- "incusos/awx-manifests/*"
|
||||||
- "notes/awx-guide.md"
|
- "notes/awx-guide.md"
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,7 +1,7 @@
|
||||||
---
|
---
|
||||||
paths:
|
paths:
|
||||||
- "bin/lab-test"
|
- "incusos/lab-test"
|
||||||
- "bin/incusos-proxmox"
|
- "incusos/incusos-proxmox"
|
||||||
- "notes/clustering-guide.md"
|
- "notes/clustering-guide.md"
|
||||||
- "notes/production-lab-guide.md"
|
- "notes/production-lab-guide.md"
|
||||||
---
|
---
|
||||||
|
|
|
||||||
|
|
@ -1,7 +1,7 @@
|
||||||
---
|
---
|
||||||
paths:
|
paths:
|
||||||
- "bin/deploy-haproxy"
|
- "incusos/deploy-haproxy"
|
||||||
- "bin/helpers/aether-browser"
|
- "incusos/helpers/aether-browser"
|
||||||
- "notes/haproxy-guide.md"
|
- "notes/haproxy-guide.md"
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,8 +1,8 @@
|
||||||
# Hetzner Setup Context
|
# Hetzner Setup Context
|
||||||
|
|
||||||
paths:
|
paths:
|
||||||
- envs/hetzner/*
|
- hetzner/*
|
||||||
- envs/hetzner/setup/*
|
- incusos/targets/hetzner/*
|
||||||
|
|
||||||
## Network Topology
|
## Network Topology
|
||||||
|
|
||||||
|
|
@ -35,7 +35,7 @@ VMs cannot use public IPs directly.
|
||||||
|
|
||||||
### proxmox-setup Script
|
### proxmox-setup Script
|
||||||
|
|
||||||
`envs/hetzner/setup/proxmox-setup` runs from the workstation over SSH.
|
`hetzner/proxmox-setup` runs from the workstation over SSH.
|
||||||
All commands executed via `ssh <host> <command>`.
|
All commands executed via `ssh <host> <command>`.
|
||||||
Follows same conventions as other scripts: setup_colors, info/warn/error,
|
Follows same conventions as other scripts: setup_colors, info/warn/error,
|
||||||
--dry-run, --yes flags.
|
--dry-run, --yes flags.
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,6 @@
|
||||||
---
|
---
|
||||||
paths:
|
paths:
|
||||||
- "bin/helpers/incusos-health"
|
- "incusos/helpers/incusos-health"
|
||||||
- "notes/incusos-break-fix.md"
|
- "notes/incusos-break-fix.md"
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,10 +1,10 @@
|
||||||
---
|
---
|
||||||
paths:
|
paths:
|
||||||
- "bin/incusos-iso"
|
- "incusos/incusos-iso"
|
||||||
- "bin/incusos-seed"
|
- "incusos/incusos-seed"
|
||||||
- "bin/incusos-proxmox"
|
- "incusos/incusos-proxmox"
|
||||||
- "bin/helpers/*"
|
- "incusos/helpers/*"
|
||||||
- "bin/examples/*"
|
- "incusos/examples/*"
|
||||||
---
|
---
|
||||||
|
|
||||||
# IncusOS Scripts Context
|
# IncusOS Scripts Context
|
||||||
|
|
|
||||||
|
|
@ -1,8 +1,8 @@
|
||||||
---
|
---
|
||||||
paths:
|
paths:
|
||||||
- "bin/incusos-proxmox"
|
- "incusos/incusos-proxmox"
|
||||||
- "bin/lab-test"
|
- "incusos/lab-test"
|
||||||
- "bin/examples/*"
|
- "incusos/examples/*"
|
||||||
---
|
---
|
||||||
|
|
||||||
# Lab Infrastructure
|
# Lab Infrastructure
|
||||||
|
|
|
||||||
|
|
@ -1,7 +1,7 @@
|
||||||
---
|
---
|
||||||
paths:
|
paths:
|
||||||
- "bin/deploy-observability"
|
- "incusos/deploy-observability"
|
||||||
- "bin/data/observability-dashboards"
|
- "incusos/observability-dashboards"
|
||||||
- "notes/observability-guide.md"
|
- "notes/observability-guide.md"
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,8 +1,8 @@
|
||||||
---
|
---
|
||||||
paths:
|
paths:
|
||||||
- "notes/operations-center-guide.md"
|
- "notes/operations-center-guide.md"
|
||||||
- "bin/incusos-proxmox"
|
- "incusos/incusos-proxmox"
|
||||||
- "bin/examples/*oc*"
|
- "incusos/examples/*oc*"
|
||||||
---
|
---
|
||||||
|
|
||||||
# Operations Center
|
# Operations Center
|
||||||
|
|
|
||||||
|
|
@ -1,8 +1,8 @@
|
||||||
# OVN Internals Context
|
# OVN Internals Context
|
||||||
|
|
||||||
paths:
|
paths:
|
||||||
- bin/helpers/ovn-inspect
|
- ovn-inspect
|
||||||
- notes/ovn-deep-dive*
|
- ovn-deep-dive
|
||||||
|
|
||||||
## OVN Architecture in this Cluster
|
## OVN Architecture in this Cluster
|
||||||
|
|
||||||
|
|
@ -26,7 +26,7 @@ Incus manages OVN objects using a `net8` prefix convention.
|
||||||
- NB/SB commands: `incus exec oc-node-01:ovn-central -- ovn-nbctl ...`
|
- NB/SB commands: `incus exec oc-node-01:ovn-central -- ovn-nbctl ...`
|
||||||
- OVS commands: Need privileged container with `/run/openvswitch` mounted
|
- OVS commands: Need privileged container with `/run/openvswitch` mounted
|
||||||
(deploy Alpine on net-prod, `apk add openvswitch`)
|
(deploy Alpine on net-prod, `apk add openvswitch`)
|
||||||
- Helper script: `bin/helpers/ovn-inspect --nb|--sb|--ovs|--full|--trace`
|
- Helper script: `incusos/helpers/ovn-inspect --nb|--sb|--ovs|--full|--trace`
|
||||||
|
|
||||||
### Important Details
|
### Important Details
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,10 +1,10 @@
|
||||||
---
|
---
|
||||||
paths:
|
paths:
|
||||||
- "bin/incusos-proxmox"
|
- "incusos/incusos-proxmox"
|
||||||
- "bin/diagnostic/observe-deploy"
|
- "incusos/observe-deploy"
|
||||||
- "bin/helpers/*"
|
- "incusos/helpers/*"
|
||||||
- "bin/examples/*"
|
- "incusos/examples/*"
|
||||||
- "bin/TESTING.md"
|
- "incusos/TESTING.md"
|
||||||
---
|
---
|
||||||
|
|
||||||
# Proxmox VE Deployment
|
# Proxmox VE Deployment
|
||||||
|
|
|
||||||
|
|
@ -16,7 +16,7 @@ Make an authenticated Proxmox API call with proper token handling.
|
||||||
|
|
||||||
1. Run the helper script:
|
1. Run the helper script:
|
||||||
```bash
|
```bash
|
||||||
/home/maarten/dev/incus-contrib/bin/helpers/proxmox-api {{args}}
|
/home/maarten/dev/incus-contrib/incusos/helpers/proxmox-api {{args}}
|
||||||
```
|
```
|
||||||
2. Parse the JSON response
|
2. Parse the JSON response
|
||||||
3. Present the results in a readable format
|
3. Present the results in a readable format
|
||||||
|
|
|
||||||
|
|
@ -16,7 +16,7 @@ Take a console screenshot of a Proxmox VM and display it.
|
||||||
|
|
||||||
1. Run the helper script to capture the screenshot:
|
1. Run the helper script to capture the screenshot:
|
||||||
```bash
|
```bash
|
||||||
/home/maarten/dev/incus-contrib/bin/helpers/proxmox-screenshot {{vmid}}
|
/home/maarten/dev/incus-contrib/incusos/helpers/proxmox-screenshot {{vmid}}
|
||||||
```
|
```
|
||||||
2. Read the output path from stdout
|
2. Read the output path from stdout
|
||||||
3. Use the Read tool to view the PNG image
|
3. Use the Read tool to view the PNG image
|
||||||
|
|
|
||||||
|
|
@ -26,11 +26,13 @@ Thumbs.db
|
||||||
# Go binaries (flasher-tool)
|
# Go binaries (flasher-tool)
|
||||||
flasher-tool
|
flasher-tool
|
||||||
|
|
||||||
# Per-environment secrets and connection configs
|
# Proxmox connection config (contains credentials)
|
||||||
envs/*/env
|
proxmox.yaml
|
||||||
envs/*/proxmox.yaml
|
|
||||||
|
|
||||||
# Legacy: root env file (backward compat)
|
# Per-target env files
|
||||||
|
env-*
|
||||||
|
|
||||||
|
# Environment secrets
|
||||||
env
|
env
|
||||||
|
|
||||||
# Observational deploy output (screenshots, logs)
|
# Observational deploy output (screenshots, logs)
|
||||||
|
|
@ -41,6 +43,3 @@ sources/
|
||||||
|
|
||||||
# Playwright MCP logs
|
# Playwright MCP logs
|
||||||
.playwright-mcp/
|
.playwright-mcp/
|
||||||
|
|
||||||
# Claude Code local state
|
|
||||||
.claude/scheduled_tasks.lock
|
|
||||||
|
|
|
||||||
74
CLAUDE.md
74
CLAUDE.md
|
|
@ -20,16 +20,16 @@ incus-contrib/
|
||||||
│ └── playbooks/
|
│ └── playbooks/
|
||||||
│ ├── post-deploy.yml # Runs after Aether creates an instance
|
│ ├── post-deploy.yml # Runs after Aether creates an instance
|
||||||
│ └── decommission.yml # Runs before Aether deletes an instance
|
│ └── decommission.yml # Runs before Aether deletes an instance
|
||||||
├── bin/ # ALL reusable scripts
|
├── incusos/ # IncusOS installation tooling
|
||||||
|
│ ├── README.md
|
||||||
│ ├── incusos-iso # ISO/IMG builder (wraps flasher-tool)
|
│ ├── incusos-iso # ISO/IMG builder (wraps flasher-tool)
|
||||||
│ ├── incusos-seed # Seed archive generator (Linux + macOS)
|
│ ├── incusos-seed # Seed archive generator (Linux + macOS)
|
||||||
│ ├── incusos-proxmox # Declarative Proxmox VM deployment + lab lifecycle
|
│ ├── incusos-proxmox # Declarative Proxmox VM deployment + lab lifecycle
|
||||||
│ ├── lab-test # Guided lab validation (12 test phases)
|
|
||||||
│ ├── deploy-awx # AWX deployment + management on Incus cluster
|
│ ├── deploy-awx # AWX deployment + management on Incus cluster
|
||||||
│ ├── deploy-haproxy # HAProxy LB deployment + management via Aether
|
│ ├── deploy-haproxy # HAProxy LB deployment + management via Aether
|
||||||
│ ├── deploy-observability # Prometheus + Grafana + Loki observability stack
|
│ ├── deploy-observability # Prometheus + Grafana + Loki observability stack
|
||||||
│ ├── manage-dashboards # Grafana dashboard management
|
│ ├── awx-manifests/ # K8s manifests for AWX Operator + instance
|
||||||
│ ├── pve-api # Python Proxmox API helper
|
│ ├── observability-dashboards/ # Grafana dashboard JSON exports
|
||||||
│ ├── helpers/
|
│ ├── helpers/
|
||||||
│ │ ├── proxmox-screenshot # VMID -> PNG console screenshot
|
│ │ ├── proxmox-screenshot # VMID -> PNG console screenshot
|
||||||
│ │ ├── proxmox-api # Authenticated API calls (handles ! in token)
|
│ │ ├── proxmox-api # Authenticated API calls (handles ! in token)
|
||||||
|
|
@ -37,41 +37,18 @@ incus-contrib/
|
||||||
│ │ ├── ovn-inspect # OVN/OVS topology inspection (--nb/--sb/--ovs/--trace)
|
│ │ ├── ovn-inspect # OVN/OVS topology inspection (--nb/--sb/--ovs/--trace)
|
||||||
│ │ ├── incus-mitm # API traffic capture and analysis (--capture/--live/--analyze)
|
│ │ ├── incus-mitm # API traffic capture and analysis (--capture/--live/--analyze)
|
||||||
│ │ └── incusos-health # IncusOS health inspection (--status/--tpm/--partitions/--all)
|
│ │ └── incusos-health # IncusOS health inspection (--status/--tpm/--partitions/--all)
|
||||||
│ ├── data/ # Static data consumed by scripts
|
│ ├── lab-test # Guided lab validation (12 test phases)
|
||||||
│ │ ├── awx-manifests/ # K8s manifests for AWX Operator + instance
|
│ ├── observe-deploy # Single-VM deploy with console screenshots
|
||||||
│ │ └── observability-dashboards/ # Grafana dashboard JSON exports
|
│ ├── targets/ # Per-host configs (beelink/, hetzner/)
|
||||||
│ ├── diagnostic/ # Investigation/one-off tools
|
│ ├── proxmox.yaml # Proxmox connection (gitignored)
|
||||||
│ │ ├── observe-deploy # Single-VM deploy with console screenshots
|
│ ├── TESTING.md
|
||||||
│ │ ├── investigate-boot # Boot investigation
|
│ └── examples/ # Example seed + Proxmox YAML files
|
||||||
│ │ └── test-boot-reliability # Parallel boot testing
|
├── hetzner/ # Hetzner dedicated server setup
|
||||||
│ ├── examples/ # Example seed + Proxmox YAML files
|
│ ├── README.md # Overview, prerequisites, quickstart
|
||||||
│ └── TESTING.md
|
│ ├── hetzner-setup.md # Step-by-step Proxmox setup guide
|
||||||
├── envs/ # Per-environment config + secrets
|
│ └── proxmox-setup # Interactive setup helper (runs over SSH)
|
||||||
│ ├── README.md # How environments work
|
├── env # PROXMOX_TOKEN_SECRET, PROXMOX_ROOT_PASSWORD, AETHER_ADMIN_PASSWORD (gitignored)
|
||||||
│ ├── beelink/ # Beelink mini PC target
|
└── notes/ # Reference guides (clustering, networking, OC, AWX, etc.)
|
||||||
│ │ ├── env # Secrets (gitignored)
|
|
||||||
│ │ ├── proxmox.yaml # Proxmox connection (gitignored)
|
|
||||||
│ │ ├── proxmox.yaml.example
|
|
||||||
│ │ ├── lab-cluster.yaml
|
|
||||||
│ │ ├── awx.yaml
|
|
||||||
│ │ ├── haproxy.yaml
|
|
||||||
│ │ └── observability.yaml
|
|
||||||
│ └── hetzner/ # Hetzner dedicated server target
|
|
||||||
│ ├── env # Secrets (gitignored)
|
|
||||||
│ ├── proxmox.yaml # Proxmox connection (gitignored)
|
|
||||||
│ ├── proxmox.yaml.example
|
|
||||||
│ ├── lab-cluster.yaml
|
|
||||||
│ ├── lab-production.yaml
|
|
||||||
│ ├── awx.yaml
|
|
||||||
│ ├── haproxy.yaml
|
|
||||||
│ ├── observability.yaml
|
|
||||||
│ └── setup/ # Host provisioning
|
|
||||||
│ ├── README.md
|
|
||||||
│ ├── hetzner-setup.md
|
|
||||||
│ ├── hetzner-lab-guide.md
|
|
||||||
│ └── proxmox-setup
|
|
||||||
├── notes/ # Reference guides (clustering, networking, OC, AWX, etc.)
|
|
||||||
└── sources/ # External software (gitignored)
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## Capabilities you have
|
## Capabilities you have
|
||||||
|
|
@ -79,7 +56,7 @@ incus-contrib/
|
||||||
### Proxmox VM screenshots
|
### Proxmox VM screenshots
|
||||||
Take console screenshots during deploys to see boot progress or errors:
|
Take console screenshots during deploys to see boot progress or errors:
|
||||||
```
|
```
|
||||||
bin/helpers/proxmox-screenshot VMID [/tmp/output.png]
|
incusos/helpers/proxmox-screenshot VMID [/tmp/output.png]
|
||||||
```
|
```
|
||||||
Use the Read tool on the PNG to view it. **USE THIS PROACTIVELY** during
|
Use the Read tool on the PNG to view it. **USE THIS PROACTIVELY** during
|
||||||
deploy scenarios to monitor boot progress, diagnose hangs, and verify installs.
|
deploy scenarios to monitor boot progress, diagnose hangs, and verify installs.
|
||||||
|
|
@ -87,9 +64,9 @@ deploy scenarios to monitor boot progress, diagnose hangs, and verify installs.
|
||||||
### Proxmox API calls
|
### Proxmox API calls
|
||||||
Make authenticated API calls (handles the `!` in `automation@pve!deploy`):
|
Make authenticated API calls (handles the `!` in `automation@pve!deploy`):
|
||||||
```
|
```
|
||||||
bin/helpers/proxmox-api GET /nodes/pve/qemu/900/status/current --json
|
incusos/helpers/proxmox-api GET /nodes/pve/qemu/900/status/current --json
|
||||||
bin/helpers/proxmox-api GET /nodes/pve/status --json
|
incusos/helpers/proxmox-api GET /nodes/pve/status --json
|
||||||
bin/helpers/proxmox-api POST /nodes/pve/qemu/900/status/stop
|
incusos/helpers/proxmox-api POST /nodes/pve/qemu/900/status/stop
|
||||||
```
|
```
|
||||||
|
|
||||||
### Aether API
|
### Aether API
|
||||||
|
|
@ -98,7 +75,7 @@ Aether has a documented REST API at `https://192.168.102.160:8443/api/`.
|
||||||
- **Swagger UI**: `https://192.168.102.160:8443/api/docs`
|
- **Swagger UI**: `https://192.168.102.160:8443/api/docs`
|
||||||
- **OpenAPI spec**: `https://192.168.102.160:8443/api/swagger.yaml`
|
- **OpenAPI spec**: `https://192.168.102.160:8443/api/swagger.yaml`
|
||||||
- **Auth**: JWT via `POST /api/auth/token` with `{"username":"...","password":"..."}`
|
- **Auth**: JWT via `POST /api/auth/token` with `{"username":"...","password":"..."}`
|
||||||
Credentials in `envs/*/env` file (`AETHER_ADMIN_PASSWORD`). Token valid 24h.
|
Credentials in `env` file (`AETHER_ADMIN_PASSWORD`). Token valid 24h.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Get JWT
|
# Get JWT
|
||||||
|
|
@ -140,11 +117,11 @@ that curl cannot handle reliably. **USE PLAYWRIGHT** for these interactions.
|
||||||
`browser_click`, `browser_fill`, `browser_screenshot` etc. as tools when
|
`browser_click`, `browser_fill`, `browser_screenshot` etc. as tools when
|
||||||
the Playwright MCP server is running. Prefer MCP tools when available.
|
the Playwright MCP server is running. Prefer MCP tools when available.
|
||||||
|
|
||||||
**Helper script**: `bin/helpers/aether-browser` provides standalone
|
**Helper script**: `incusos/helpers/aether-browser` provides standalone
|
||||||
Playwright automation when MCP tools aren't loaded:
|
Playwright automation when MCP tools aren't loaded:
|
||||||
```bash
|
```bash
|
||||||
source envs/beelink/env # loads AETHER_ADMIN_PASSWORD
|
source env # loads AETHER_ADMIN_PASSWORD
|
||||||
NODE_PATH=~/node_modules node bin/helpers/aether-browser <action> [args]
|
NODE_PATH=~/node_modules node incusos/helpers/aether-browser <action> [args]
|
||||||
```
|
```
|
||||||
|
|
||||||
Actions:
|
Actions:
|
||||||
|
|
@ -200,7 +177,6 @@ still UI-only. The API endpoint may have been added since the guide was written.
|
||||||
- **Flags**: both short (`-d`) and long (`--defaults`)
|
- **Flags**: both short (`-d`) and long (`--defaults`)
|
||||||
- **Defaults**: useful behavior with zero flags
|
- **Defaults**: useful behavior with zero flags
|
||||||
- **Dry run**: all scripts support `--dry-run`
|
- **Dry run**: all scripts support `--dry-run`
|
||||||
- **Config**: deploy scripts use `--config FILE` for environment-specific settings
|
|
||||||
- **Cert detection**: files on disk first (`~/.config/incus/client.crt`), CLI second
|
- **Cert detection**: files on disk first (`~/.config/incus/client.crt`), CLI second
|
||||||
- **Errors**: include actionable remediation steps
|
- **Errors**: include actionable remediation steps
|
||||||
- **No hardcoded package managers**: say "install the Incus client" with a link
|
- **No hardcoded package managers**: say "install the Incus client" with a link
|
||||||
|
|
@ -225,7 +201,7 @@ which files are being edited:
|
||||||
| `incusos-immutability.md` | incusos-health, incusos-break-fix |
|
| `incusos-immutability.md` | incusos-health, incusos-break-fix |
|
||||||
| `proxmox-ssh-rules.md` | **Always loaded** (no paths filter) |
|
| `proxmox-ssh-rules.md` | **Always loaded** (no paths filter) |
|
||||||
| `lab-infrastructure.md` | incusos-proxmox, lab-test, examples |
|
| `lab-infrastructure.md` | incusos-proxmox, lab-test, examples |
|
||||||
| `hetzner-setup.md` | envs/hetzner/, setup scripts |
|
| `hetzner-setup.md` | hetzner/, incusos/targets/hetzner/ |
|
||||||
|
|
||||||
For deep reference, see the guides in `notes/`.
|
For deep reference, see the guides in `notes/`.
|
||||||
|
|
||||||
|
|
|
||||||
65
README.md
65
README.md
|
|
@ -1,4 +1,4 @@
|
||||||
# incus-contrib
|
# incu-contrib
|
||||||
|
|
||||||
Peripheral tools, snippets, and scripts for working with [Incus](https://linuxcontainers.org/incus/),
|
Peripheral tools, snippets, and scripts for working with [Incus](https://linuxcontainers.org/incus/),
|
||||||
[IncusOS](https://linuxcontainers.org/incus-os/), and the broader ecosystem.
|
[IncusOS](https://linuxcontainers.org/incus-os/), and the broader ecosystem.
|
||||||
|
|
@ -7,10 +7,9 @@ Primarily aimed at home lab environments but useful in production as well.
|
||||||
|
|
||||||
## Contents
|
## Contents
|
||||||
|
|
||||||
### [`bin/`](bin/)
|
### [`incusos/`](incusos/)
|
||||||
|
|
||||||
Reusable scripts for building IncusOS installation media, deploying labs,
|
Scripts for building IncusOS installation media and seed configurations:
|
||||||
and managing services:
|
|
||||||
|
|
||||||
- **`incusos-iso`** -- Build customized IncusOS ISO/IMG images using the official flasher-tool.
|
- **`incusos-iso`** -- Build customized IncusOS ISO/IMG images using the official flasher-tool.
|
||||||
Supports both architectures (x86_64, aarch64), multiple applications (Incus, Operations Center),
|
Supports both architectures (x86_64, aarch64), multiple applications (Incus, Operations Center),
|
||||||
|
|
@ -24,29 +23,15 @@ and managing services:
|
||||||
YAML, and the script handles seed generation, ISO upload, VM creation with correct
|
YAML, and the script handles seed generation, ISO upload, VM creation with correct
|
||||||
settings (UEFI, TPM 2.0, VirtIO-SCSI), and automated installation.
|
settings (UEFI, TPM 2.0, VirtIO-SCSI), and automated installation.
|
||||||
|
|
||||||
|
- **`targets/`** -- Per-host Proxmox connection configs and lab definitions. Each
|
||||||
|
subdirectory (beelink, hetzner) has a `proxmox.yaml.example` and lab YAML files
|
||||||
|
sized for that host. See [`targets/README.md`](incusos/targets/README.md).
|
||||||
|
|
||||||
- **`deploy-awx`** -- Deploy and manage AWX (Ansible automation) on the Incus
|
- **`deploy-awx`** -- Deploy and manage AWX (Ansible automation) on the Incus
|
||||||
cluster. Handles VM creation, K3s install, AWX Operator deployment, project/template
|
cluster. Handles VM creation, K3s install, AWX Operator deployment, project/template
|
||||||
configuration, and Aether integration. Requires `--config` for environment settings.
|
configuration, and Aether integration.
|
||||||
|
|
||||||
- **`deploy-haproxy`** -- Deploy HA HAProxy load balancing via Aether. Requires
|
See [`incusos/README.md`](incusos/README.md) for full documentation.
|
||||||
`--config` for environment settings.
|
|
||||||
|
|
||||||
- **`deploy-observability`** -- Deploy Prometheus + Grafana + Loki monitoring stack.
|
|
||||||
Requires `--config` for environment settings.
|
|
||||||
|
|
||||||
See [`bin/README.md`](bin/README.md) for full documentation.
|
|
||||||
|
|
||||||
### [`envs/`](envs/)
|
|
||||||
|
|
||||||
Per-environment configuration, secrets, and deployment settings:
|
|
||||||
|
|
||||||
- **`beelink/`** -- Beelink mini PC target (LAN, vmbr0, VLAN 69, local-lvm)
|
|
||||||
- **`hetzner/`** -- Hetzner dedicated server target (WireGuard, vmbr1, local-zfs)
|
|
||||||
- **`setup/`** -- Host provisioning guide and interactive setup script
|
|
||||||
|
|
||||||
Each environment has: `env` (secrets), `proxmox.yaml` (connection),
|
|
||||||
lab YAML files (VM definitions), and service configs (`awx.yaml`, `haproxy.yaml`,
|
|
||||||
`observability.yaml`). See [`envs/README.md`](envs/README.md).
|
|
||||||
|
|
||||||
### [`ansible/`](ansible/)
|
### [`ansible/`](ansible/)
|
||||||
|
|
||||||
|
|
@ -58,6 +43,18 @@ API (not SSH) for all instance operations — works regardless of network topolo
|
||||||
- **`playbooks/decommission.yml`** -- Runs before Aether deletes an instance.
|
- **`playbooks/decommission.yml`** -- Runs before Aether deletes an instance.
|
||||||
Graceful service shutdown and decommission logging (best-effort).
|
Graceful service shutdown and decommission logging (best-effort).
|
||||||
|
|
||||||
|
### [`hetzner/`](hetzner/)
|
||||||
|
|
||||||
|
Setup guide and automation for deploying IncusOS labs on a Hetzner dedicated server:
|
||||||
|
|
||||||
|
- **`hetzner-setup.md`** -- Step-by-step guide: Proxmox install, private networking,
|
||||||
|
WireGuard tunnel, firewall lockdown, and API token creation.
|
||||||
|
- **`proxmox-setup`** -- Interactive helper script that automates the guide steps
|
||||||
|
over SSH. Configures repos, bridge, storage, WireGuard, firewall, and API tokens.
|
||||||
|
|
||||||
|
See also [`incusos/targets/`](incusos/targets/) for per-host connection configs
|
||||||
|
and lab definitions.
|
||||||
|
|
||||||
### [`notes/`](notes/)
|
### [`notes/`](notes/)
|
||||||
|
|
||||||
Guides and reference material (roughly in learning order):
|
Guides and reference material (roughly in learning order):
|
||||||
|
|
@ -92,27 +89,19 @@ Guides and reference material (roughly in learning order):
|
||||||
go install github.com/lxc/incus-os/incus-osd/cmd/flasher-tool@latest
|
go install github.com/lxc/incus-os/incus-osd/cmd/flasher-tool@latest
|
||||||
|
|
||||||
# Build a default IncusOS ISO with your client cert injected
|
# Build a default IncusOS ISO with your client cert injected
|
||||||
bin/incusos-iso
|
./incusos/incusos-iso
|
||||||
|
|
||||||
# Build for a specific architecture and application
|
# Build for a specific architecture and application
|
||||||
bin/incusos-iso --arch aarch64 --app operations-center --defaults
|
./incusos/incusos-iso --arch aarch64 --app operations-center --defaults
|
||||||
|
|
||||||
# Generate all common ISO variants at once
|
# Generate all common ISO variants at once
|
||||||
bin/incusos-iso --batch
|
./incusos/incusos-iso --batch
|
||||||
|
|
||||||
# Generate a standalone seed archive
|
# Generate a standalone seed archive
|
||||||
bin/incusos-seed --app incus --defaults --output my-seed.tar
|
./incusos/incusos-seed --app incus --defaults --output my-seed.tar
|
||||||
|
|
||||||
# Deploy a cluster on Hetzner
|
# Generate a SEED_DATA FAT image for external boot media
|
||||||
source envs/hetzner/env
|
./incusos/incusos-seed --format fat --app incus --defaults --output seed.img
|
||||||
bin/incusos-proxmox --proxmox envs/hetzner/proxmox.yaml envs/hetzner/lab-production.yaml
|
|
||||||
|
|
||||||
# Deploy AWX on Hetzner
|
|
||||||
bin/deploy-awx --config envs/hetzner/awx.yaml --deploy
|
|
||||||
|
|
||||||
# Deploy HAProxy on beelink
|
|
||||||
source envs/beelink/env
|
|
||||||
bin/deploy-haproxy --config envs/beelink/haproxy.yaml --deploy
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## Requirements
|
## Requirements
|
||||||
|
|
|
||||||
|
|
@ -29,7 +29,7 @@
|
||||||
|
|
||||||
vars:
|
vars:
|
||||||
# Incus cluster API (any cluster member works)
|
# Incus cluster API (any cluster member works)
|
||||||
incus_api: "https://10.10.0.111:8443"
|
incus_api: "https://192.168.102.140:8443"
|
||||||
# AWX runner mounts project at /runner/project/ during job execution
|
# AWX runner mounts project at /runner/project/ during job execution
|
||||||
incus_cert: "/runner/project/incus-client.crt"
|
incus_cert: "/runner/project/incus-client.crt"
|
||||||
incus_key: "/runner/project/incus-client.key"
|
incus_key: "/runner/project/incus-client.key"
|
||||||
|
|
|
||||||
436
bin/lab-status
436
bin/lab-status
|
|
@ -1,436 +0,0 @@
|
||||||
#!/usr/bin/env bash
|
|
||||||
set -euo pipefail
|
|
||||||
|
|
||||||
readonly VERSION="0.1.0"
|
|
||||||
readonly SCRIPT_NAME="lab-status"
|
|
||||||
readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
||||||
readonly REPO_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
|
|
||||||
readonly ENVS_DIR="${REPO_DIR}/envs"
|
|
||||||
|
|
||||||
# --- Defaults -----------------------------------------------------------------
|
|
||||||
|
|
||||||
QUIET=false
|
|
||||||
VERBOSE=false
|
|
||||||
ENV_NAME=""
|
|
||||||
NO_CHECK=false
|
|
||||||
JSON_OUTPUT=false
|
|
||||||
|
|
||||||
# Service arrays (parallel indexed)
|
|
||||||
declare -a SVC_NAMES=()
|
|
||||||
declare -a SVC_URLS=()
|
|
||||||
declare -a SVC_STATUSES=()
|
|
||||||
|
|
||||||
# --- Colors & logging --------------------------------------------------------
|
|
||||||
|
|
||||||
setup_colors() {
|
|
||||||
if [[ -t 1 ]] && [[ "${NO_COLOR:-}" != "1" ]] && [[ "${TERM:-}" != "dumb" ]]; then
|
|
||||||
RED='\033[0;31m'
|
|
||||||
GREEN='\033[0;32m'
|
|
||||||
YELLOW='\033[0;33m'
|
|
||||||
BLUE='\033[0;34m'
|
|
||||||
CYAN='\033[0;36m'
|
|
||||||
BOLD='\033[1m'
|
|
||||||
DIM='\033[2m'
|
|
||||||
RESET='\033[0m'
|
|
||||||
else
|
|
||||||
RED='' GREEN='' YELLOW='' BLUE='' CYAN='' BOLD='' DIM='' RESET=''
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
info() { [[ "$QUIET" == true ]] && return; echo -e "${BLUE}[info]${RESET} $*"; }
|
|
||||||
success() { [[ "$QUIET" == true ]] && return; echo -e "${GREEN}[ok]${RESET} $*"; }
|
|
||||||
warn() { echo -e "${YELLOW}[warn]${RESET} $*" >&2; }
|
|
||||||
error() { echo -e "${RED}[error]${RESET} $*" >&2; }
|
|
||||||
detail() { [[ "$VERBOSE" != true ]] && return; echo -e "${DIM} $*${RESET}"; }
|
|
||||||
|
|
||||||
# --- Usage --------------------------------------------------------------------
|
|
||||||
|
|
||||||
usage() {
|
|
||||||
cat <<EOF
|
|
||||||
${BOLD}${SCRIPT_NAME}${RESET} v${VERSION} - Lab service status overview
|
|
||||||
|
|
||||||
${BOLD}USAGE${RESET}
|
|
||||||
${SCRIPT_NAME} [OPTIONS]
|
|
||||||
|
|
||||||
${BOLD}OPTIONS${RESET}
|
|
||||||
-e, --env NAME Environment name (auto-detects if only one exists)
|
|
||||||
--no-check Skip reachability checks, just show URLs
|
|
||||||
--json Machine-readable JSON output
|
|
||||||
-v, --verbose Show extra details
|
|
||||||
-q, --quiet Suppress info messages
|
|
||||||
-h, --help Show this help
|
|
||||||
|
|
||||||
${BOLD}DESCRIPTION${RESET}
|
|
||||||
Reads YAML configs from envs/<env>/ and checks whether each service
|
|
||||||
is reachable. Any HTTP response (even 401/403) counts as UP; connection
|
|
||||||
timeout or refused counts as DOWN.
|
|
||||||
|
|
||||||
Services detected: Proxmox, OC Server, Cluster Nodes, Aether, AWX,
|
|
||||||
HAProxy VIP, Grafana, Prometheus.
|
|
||||||
|
|
||||||
${BOLD}EXAMPLES${RESET}
|
|
||||||
# Show hetzner lab with reachability checks
|
|
||||||
${SCRIPT_NAME} --env hetzner
|
|
||||||
|
|
||||||
# Show beelink lab without checks
|
|
||||||
${SCRIPT_NAME} --env beelink --no-check
|
|
||||||
|
|
||||||
# JSON output for scripting
|
|
||||||
${SCRIPT_NAME} --env hetzner --json
|
|
||||||
|
|
||||||
EOF
|
|
||||||
}
|
|
||||||
|
|
||||||
# --- YAML helpers -------------------------------------------------------------
|
|
||||||
|
|
||||||
# Extract a value from a YAML section: yaml_val FILE SECTION KEY
|
|
||||||
yaml_val() {
|
|
||||||
local file="$1" section="$2" key="$3"
|
|
||||||
grep -A50 "^${section}:" "$file" \
|
|
||||||
| grep "^ *${key}:" | head -1 \
|
|
||||||
| sed "s/^ *${key}: *//" | tr -d '"' | sed 's/ *#.*//' || true
|
|
||||||
}
|
|
||||||
|
|
||||||
# Extract a top-level YAML value: yaml_val_top FILE KEY
|
|
||||||
yaml_val_top() {
|
|
||||||
local file="$1" key="$2"
|
|
||||||
grep "^${key}:" "$file" | head -1 \
|
|
||||||
| sed "s/^${key}: *//" | tr -d '"' | sed 's/ *#.*//' || true
|
|
||||||
}
|
|
||||||
|
|
||||||
# --- Service collection -------------------------------------------------------
|
|
||||||
|
|
||||||
add_service() {
|
|
||||||
local name="$1" url="$2"
|
|
||||||
SVC_NAMES+=("$name")
|
|
||||||
SVC_URLS+=("$url")
|
|
||||||
SVC_STATUSES+=("")
|
|
||||||
}
|
|
||||||
|
|
||||||
detect_env() {
|
|
||||||
if [[ -n "$ENV_NAME" ]]; then
|
|
||||||
if [[ ! -d "${ENVS_DIR}/${ENV_NAME}" ]]; then
|
|
||||||
error "Environment not found: ${ENV_NAME}"
|
|
||||||
local available
|
|
||||||
available=$(ls "$ENVS_DIR" 2>/dev/null | tr '\n' ' ')
|
|
||||||
[[ -n "$available" ]] && error "Available: ${available}"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
return
|
|
||||||
fi
|
|
||||||
|
|
||||||
local envs=()
|
|
||||||
for d in "$ENVS_DIR"/*/; do
|
|
||||||
[[ -d "$d" ]] && envs+=("$(basename "$d")")
|
|
||||||
done
|
|
||||||
|
|
||||||
if [[ ${#envs[@]} -eq 1 ]]; then
|
|
||||||
ENV_NAME="${envs[0]}"
|
|
||||||
info "Auto-detected environment: ${ENV_NAME}"
|
|
||||||
elif [[ ${#envs[@]} -eq 0 ]]; then
|
|
||||||
error "No environments found in ${ENVS_DIR}/"
|
|
||||||
exit 1
|
|
||||||
else
|
|
||||||
error "Multiple environments found: ${envs[*]}"
|
|
||||||
error "Use --env NAME to select one"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
collect_proxmox() {
|
|
||||||
local env_dir="${ENVS_DIR}/${ENV_NAME}"
|
|
||||||
local pve_file=""
|
|
||||||
|
|
||||||
if [[ -f "${env_dir}/proxmox.yaml" ]]; then
|
|
||||||
pve_file="${env_dir}/proxmox.yaml"
|
|
||||||
elif [[ -f "${env_dir}/proxmox.yaml.example" ]]; then
|
|
||||||
pve_file="${env_dir}/proxmox.yaml.example"
|
|
||||||
warn "Using proxmox.yaml.example (proxmox.yaml not found)"
|
|
||||||
else
|
|
||||||
detail "No proxmox.yaml found, skipping Proxmox"
|
|
||||||
return
|
|
||||||
fi
|
|
||||||
|
|
||||||
local host
|
|
||||||
host=$(yaml_val_top "$pve_file" "host")
|
|
||||||
if [[ -n "$host" ]]; then
|
|
||||||
host="${host%%/*}"
|
|
||||||
add_service "Proxmox" "https://${host}:8006"
|
|
||||||
detail "Proxmox from $(basename "$pve_file"): ${host}"
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
collect_lab_vms() {
|
|
||||||
local env_dir="${ENVS_DIR}/${ENV_NAME}"
|
|
||||||
local lab_file=""
|
|
||||||
|
|
||||||
# Prefer lab-production.yaml over lab-cluster.yaml
|
|
||||||
if [[ -f "${env_dir}/lab-production.yaml" ]]; then
|
|
||||||
lab_file="${env_dir}/lab-production.yaml"
|
|
||||||
elif [[ -f "${env_dir}/lab-cluster.yaml" ]]; then
|
|
||||||
lab_file="${env_dir}/lab-cluster.yaml"
|
|
||||||
else
|
|
||||||
detail "No lab-*.yaml found, skipping cluster VMs"
|
|
||||||
return
|
|
||||||
fi
|
|
||||||
|
|
||||||
detail "Lab VMs from $(basename "$lab_file")"
|
|
||||||
|
|
||||||
local vm_data
|
|
||||||
vm_data=$(python3 -c "
|
|
||||||
import sys, re
|
|
||||||
|
|
||||||
try:
|
|
||||||
import yaml
|
|
||||||
with open(sys.argv[1]) as f:
|
|
||||||
data = yaml.safe_load(f)
|
|
||||||
except ImportError:
|
|
||||||
data = {'vms': []}
|
|
||||||
with open(sys.argv[1]) as f:
|
|
||||||
content = f.read()
|
|
||||||
in_vms = False
|
|
||||||
current = {}
|
|
||||||
for line in content.split('\n'):
|
|
||||||
if line.startswith('vms:'):
|
|
||||||
in_vms = True
|
|
||||||
continue
|
|
||||||
if in_vms:
|
|
||||||
if line.startswith(' - '):
|
|
||||||
if current:
|
|
||||||
data['vms'].append(current)
|
|
||||||
current = {}
|
|
||||||
line = line[4:]
|
|
||||||
elif line.startswith(' '):
|
|
||||||
line = line[4:]
|
|
||||||
elif line and not line.startswith(' ') and not line.startswith('#'):
|
|
||||||
if current:
|
|
||||||
data['vms'].append(current)
|
|
||||||
break
|
|
||||||
else:
|
|
||||||
continue
|
|
||||||
m = re.match(r'(\w+):\s*(.*)', line.strip())
|
|
||||||
if m:
|
|
||||||
current[m.group(1)] = m.group(2).strip('\"')
|
|
||||||
if current:
|
|
||||||
data['vms'].append(current)
|
|
||||||
|
|
||||||
for vm in data.get('vms', []):
|
|
||||||
name = vm.get('name', '')
|
|
||||||
app = vm.get('app', '')
|
|
||||||
ip = vm.get('ip', '').split('/')[0]
|
|
||||||
print(f'{name}|{app}|{ip}')
|
|
||||||
" "$lab_file" 2>/dev/null) || return
|
|
||||||
|
|
||||||
local node_idx=0
|
|
||||||
while IFS='|' read -r name app ip; do
|
|
||||||
[[ -z "$ip" ]] && continue
|
|
||||||
case "$app" in
|
|
||||||
operations-center)
|
|
||||||
add_service "OC Server" "https://${ip}:8443"
|
|
||||||
;;
|
|
||||||
incus)
|
|
||||||
node_idx=$((node_idx + 1))
|
|
||||||
local label
|
|
||||||
label=$(printf "Cluster Node %02d" "$node_idx")
|
|
||||||
add_service "$label" "https://${ip}:8443"
|
|
||||||
;;
|
|
||||||
esac
|
|
||||||
done <<< "$vm_data"
|
|
||||||
}
|
|
||||||
|
|
||||||
collect_haproxy() {
|
|
||||||
local env_dir="${ENVS_DIR}/${ENV_NAME}"
|
|
||||||
local file="${env_dir}/haproxy.yaml"
|
|
||||||
[[ -f "$file" ]] || return
|
|
||||||
|
|
||||||
local aether_url vip
|
|
||||||
aether_url=$(yaml_val "$file" "haproxy" "aether_url")
|
|
||||||
vip=$(yaml_val "$file" "haproxy" "vip")
|
|
||||||
|
|
||||||
if [[ -n "$aether_url" ]]; then
|
|
||||||
add_service "Aether" "$aether_url"
|
|
||||||
detail "Aether from haproxy.yaml: ${aether_url}"
|
|
||||||
fi
|
|
||||||
if [[ -n "$vip" ]]; then
|
|
||||||
vip="${vip%%/*}"
|
|
||||||
add_service "HAProxy VIP" "http://${vip}:80"
|
|
||||||
detail "HAProxy VIP from haproxy.yaml: ${vip}"
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
collect_awx() {
|
|
||||||
local env_dir="${ENVS_DIR}/${ENV_NAME}"
|
|
||||||
local file="${env_dir}/awx.yaml"
|
|
||||||
[[ -f "$file" ]] || return
|
|
||||||
|
|
||||||
local ip
|
|
||||||
ip=$(yaml_val "$file" "awx" "ip")
|
|
||||||
[[ -z "$ip" ]] && return
|
|
||||||
ip="${ip%%/*}"
|
|
||||||
add_service "AWX" "http://${ip}:30080"
|
|
||||||
detail "AWX from awx.yaml: ${ip}"
|
|
||||||
}
|
|
||||||
|
|
||||||
collect_observability() {
|
|
||||||
local env_dir="${ENVS_DIR}/${ENV_NAME}"
|
|
||||||
local file="${env_dir}/observability.yaml"
|
|
||||||
[[ -f "$file" ]] || return
|
|
||||||
|
|
||||||
local forward_vip
|
|
||||||
forward_vip=$(yaml_val "$file" "observability" "forward_vip")
|
|
||||||
[[ -z "$forward_vip" ]] && return
|
|
||||||
forward_vip="${forward_vip%%/*}"
|
|
||||||
|
|
||||||
add_service "Grafana" "http://${forward_vip}:3000"
|
|
||||||
add_service "Prometheus" "http://${forward_vip}:9090"
|
|
||||||
detail "Observability from observability.yaml: forward_vip=${forward_vip}"
|
|
||||||
}
|
|
||||||
|
|
||||||
# --- Reachability checks -----------------------------------------------------
|
|
||||||
|
|
||||||
check_reachability() {
|
|
||||||
local url="$1" result_file="$2"
|
|
||||||
local http_code
|
|
||||||
http_code=$(curl -sk --connect-timeout 2 --max-time 3 \
|
|
||||||
-o /dev/null -w '%{http_code}' "$url" 2>/dev/null) || http_code="000"
|
|
||||||
if [[ "$http_code" != "000" ]]; then
|
|
||||||
echo "UP" > "$result_file"
|
|
||||||
else
|
|
||||||
echo "DOWN" > "$result_file"
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
run_checks() {
|
|
||||||
local tmpdir
|
|
||||||
tmpdir=$(mktemp -d "${TMPDIR:-/tmp}/${SCRIPT_NAME}-XXXXXX")
|
|
||||||
# shellcheck disable=SC2064
|
|
||||||
trap "rm -rf '$tmpdir'" EXIT
|
|
||||||
|
|
||||||
local i
|
|
||||||
for i in "${!SVC_URLS[@]}"; do
|
|
||||||
check_reachability "${SVC_URLS[$i]}" "${tmpdir}/${i}" &
|
|
||||||
done
|
|
||||||
wait
|
|
||||||
|
|
||||||
for i in "${!SVC_URLS[@]}"; do
|
|
||||||
if [[ -f "${tmpdir}/${i}" ]]; then
|
|
||||||
SVC_STATUSES[$i]=$(cat "${tmpdir}/${i}")
|
|
||||||
else
|
|
||||||
SVC_STATUSES[$i]="DOWN"
|
|
||||||
fi
|
|
||||||
done
|
|
||||||
}
|
|
||||||
|
|
||||||
# --- Output -------------------------------------------------------------------
|
|
||||||
|
|
||||||
print_table() {
|
|
||||||
local max_name=7 max_url=3 # minimum: "Service" / "URL"
|
|
||||||
local i
|
|
||||||
for i in "${!SVC_NAMES[@]}"; do
|
|
||||||
[[ ${#SVC_NAMES[$i]} -gt $max_name ]] && max_name=${#SVC_NAMES[$i]}
|
|
||||||
[[ ${#SVC_URLS[$i]} -gt $max_url ]] && max_url=${#SVC_URLS[$i]}
|
|
||||||
done
|
|
||||||
|
|
||||||
local total_width=$((max_name + max_url + 14))
|
|
||||||
local sep
|
|
||||||
sep=$(printf '%*s' "$total_width" '' | tr ' ' '─')
|
|
||||||
|
|
||||||
echo
|
|
||||||
echo -e "${BOLD}Lab Environment: ${ENV_NAME}${RESET}"
|
|
||||||
echo "$sep"
|
|
||||||
printf "%-${max_name}s %-${max_url}s %s\n" "Service" "URL" "Status"
|
|
||||||
echo "$sep"
|
|
||||||
|
|
||||||
for i in "${!SVC_NAMES[@]}"; do
|
|
||||||
local status="${SVC_STATUSES[$i]:-—}"
|
|
||||||
local status_colored
|
|
||||||
case "$status" in
|
|
||||||
UP) status_colored="${GREEN}UP${RESET}" ;;
|
|
||||||
DOWN) status_colored="${RED}DOWN${RESET}" ;;
|
|
||||||
*) status_colored="$status" ;;
|
|
||||||
esac
|
|
||||||
printf "%-${max_name}s %-${max_url}s " "${SVC_NAMES[$i]}" "${SVC_URLS[$i]}"
|
|
||||||
echo -e "$status_colored"
|
|
||||||
done
|
|
||||||
|
|
||||||
echo "$sep"
|
|
||||||
echo
|
|
||||||
}
|
|
||||||
|
|
||||||
print_json() {
|
|
||||||
local last=$((${#SVC_NAMES[@]} - 1))
|
|
||||||
echo "["
|
|
||||||
local i
|
|
||||||
for i in "${!SVC_NAMES[@]}"; do
|
|
||||||
local comma=","
|
|
||||||
[[ $i -eq $last ]] && comma=""
|
|
||||||
local status_json
|
|
||||||
if [[ -n "${SVC_STATUSES[$i]:-}" ]]; then
|
|
||||||
status_json="\"${SVC_STATUSES[$i]}\""
|
|
||||||
else
|
|
||||||
status_json="null"
|
|
||||||
fi
|
|
||||||
printf ' {"service": "%s", "url": "%s", "status": %s}%s\n' \
|
|
||||||
"${SVC_NAMES[$i]}" "${SVC_URLS[$i]}" "$status_json" "$comma"
|
|
||||||
done
|
|
||||||
echo "]"
|
|
||||||
}
|
|
||||||
|
|
||||||
# --- Argument parsing ---------------------------------------------------------
|
|
||||||
|
|
||||||
parse_args() {
|
|
||||||
while [[ $# -gt 0 ]]; do
|
|
||||||
case "$1" in
|
|
||||||
-e|--env)
|
|
||||||
[[ $# -lt 2 ]] && { error "--env requires a value"; exit 1; }
|
|
||||||
ENV_NAME="$2"; shift
|
|
||||||
;;
|
|
||||||
--no-check) NO_CHECK=true ;;
|
|
||||||
--json) JSON_OUTPUT=true ;;
|
|
||||||
-v|--verbose) VERBOSE=true ;;
|
|
||||||
-q|--quiet) QUIET=true ;;
|
|
||||||
-h|--help) usage; exit 0 ;;
|
|
||||||
*)
|
|
||||||
error "Unknown option: $1"
|
|
||||||
echo "Run '${SCRIPT_NAME} --help' for usage."
|
|
||||||
exit 1
|
|
||||||
;;
|
|
||||||
esac
|
|
||||||
shift
|
|
||||||
done
|
|
||||||
}
|
|
||||||
|
|
||||||
# --- Main ---------------------------------------------------------------------
|
|
||||||
|
|
||||||
main() {
|
|
||||||
setup_colors
|
|
||||||
parse_args "$@"
|
|
||||||
detect_env
|
|
||||||
|
|
||||||
# Collect services from all config files
|
|
||||||
collect_proxmox
|
|
||||||
collect_lab_vms
|
|
||||||
collect_haproxy
|
|
||||||
collect_awx
|
|
||||||
collect_observability
|
|
||||||
|
|
||||||
if [[ ${#SVC_NAMES[@]} -eq 0 ]]; then
|
|
||||||
warn "No services found in envs/${ENV_NAME}/"
|
|
||||||
exit 0
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Run reachability checks (parallel)
|
|
||||||
if [[ "$NO_CHECK" != true ]]; then
|
|
||||||
info "Checking ${#SVC_NAMES[@]} services..."
|
|
||||||
run_checks
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Output
|
|
||||||
if [[ "$JSON_OUTPUT" == true ]]; then
|
|
||||||
print_json
|
|
||||||
else
|
|
||||||
print_table
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
main "$@"
|
|
||||||
|
|
@ -1,82 +0,0 @@
|
||||||
# Environment Configurations
|
|
||||||
|
|
||||||
Per-environment connection configs, secrets, and deployment settings. Each
|
|
||||||
subdirectory represents a physical host (or host type) with its own network
|
|
||||||
topology, storage backend, and resource sizing.
|
|
||||||
|
|
||||||
## Directory layout
|
|
||||||
|
|
||||||
```
|
|
||||||
envs/
|
|
||||||
├── beelink/
|
|
||||||
│ ├── env # Secrets: PROXMOX_TOKEN_SECRET, etc. (gitignored)
|
|
||||||
│ ├── proxmox.yaml # Proxmox connection (gitignored)
|
|
||||||
│ ├── proxmox.yaml.example # Connection template (LAN, vmbr0, local-lvm)
|
|
||||||
│ ├── lab-cluster.yaml # 3-node cluster (4C/4G/50G)
|
|
||||||
│ ├── awx.yaml # AWX deployment settings
|
|
||||||
│ ├── haproxy.yaml # HAProxy deployment settings
|
|
||||||
│ └── observability.yaml # Monitoring stack settings
|
|
||||||
└── hetzner/
|
|
||||||
├── env # Secrets (gitignored)
|
|
||||||
├── proxmox.yaml # Proxmox connection (gitignored)
|
|
||||||
├── proxmox.yaml.example # Connection template (WireGuard, vmbr1, local-zfs)
|
|
||||||
├── lab-cluster.yaml # 3-node cluster (8C/16G/100G)
|
|
||||||
├── lab-production.yaml # OC + 3-node cluster
|
|
||||||
├── awx.yaml # AWX deployment settings
|
|
||||||
├── haproxy.yaml # HAProxy deployment settings
|
|
||||||
├── observability.yaml # Monitoring stack settings
|
|
||||||
└── setup/ # Host provisioning (Proxmox install guide + script)
|
|
||||||
├── README.md
|
|
||||||
├── hetzner-setup.md
|
|
||||||
├── hetzner-lab-guide.md
|
|
||||||
└── proxmox-setup
|
|
||||||
```
|
|
||||||
|
|
||||||
## How to use
|
|
||||||
|
|
||||||
### 1. Source the environment
|
|
||||||
|
|
||||||
Each environment has its own `env` file with the same variable names.
|
|
||||||
Source the one for the host you're targeting:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
source envs/beelink/env # or: source envs/hetzner/env
|
|
||||||
```
|
|
||||||
|
|
||||||
### 2. Deploy VMs
|
|
||||||
|
|
||||||
```bash
|
|
||||||
bin/incusos-proxmox --proxmox envs/hetzner/proxmox.yaml \
|
|
||||||
envs/hetzner/lab-production.yaml
|
|
||||||
```
|
|
||||||
|
|
||||||
### 3. Deploy services with --config
|
|
||||||
|
|
||||||
```bash
|
|
||||||
bin/deploy-awx --config envs/hetzner/awx.yaml --deploy
|
|
||||||
bin/deploy-haproxy --config envs/hetzner/haproxy.yaml --deploy
|
|
||||||
bin/deploy-observability --config envs/hetzner/observability.yaml --deploy
|
|
||||||
```
|
|
||||||
|
|
||||||
## Adding a new environment
|
|
||||||
|
|
||||||
1. Create a new subdirectory: `envs/myhost/`
|
|
||||||
2. Copy `proxmox.yaml.example` from an existing env and edit
|
|
||||||
3. Create an `env` file with your secrets (same variable names)
|
|
||||||
4. Create lab YAML files sized for your hardware
|
|
||||||
5. Create `awx.yaml`, `haproxy.yaml`, `observability.yaml` with your IPs
|
|
||||||
|
|
||||||
## Key differences between environments
|
|
||||||
|
|
||||||
| Setting | Beelink | Hetzner |
|
|
||||||
|---------|---------|---------|
|
|
||||||
| Host | 192.168.1.10 (LAN) | 10.10.0.1 (WireGuard) |
|
|
||||||
| Method | ssh | api |
|
|
||||||
| Bridge | vmbr0 | vmbr1 (private) |
|
|
||||||
| Storage | local-lvm | local-zfs |
|
|
||||||
| VLAN | 69 | *(none)* |
|
|
||||||
| Gateway | 192.168.100.1 | 10.10.0.1 |
|
|
||||||
| VM IPs | 192.168.102.x/22 | 10.10.0.x/24 |
|
|
||||||
| VM cores | 4 | 8 |
|
|
||||||
| VM memory | 4-8 GiB | 16 GiB |
|
|
||||||
| VM disk | 50 GiB | 100 GiB |
|
|
||||||
|
|
@ -1,26 +0,0 @@
|
||||||
# awx.yaml - AWX deployment config for Beelink lab
|
|
||||||
#
|
|
||||||
# Deploys a Debian 12 VM with K3s + AWX Operator on oc-node-02.
|
|
||||||
# VM uses the flat VLAN 69 network for direct access.
|
|
||||||
#
|
|
||||||
# Architecture:
|
|
||||||
# awx (192.168.102.161) - K3s + AWX Operator + AWX instance
|
|
||||||
# AWX exposed on NodePort 30080 (HTTP)
|
|
||||||
#
|
|
||||||
# Usage:
|
|
||||||
# source envs/beelink/env
|
|
||||||
# bin/deploy-awx --config envs/beelink/awx.yaml --deploy
|
|
||||||
|
|
||||||
awx:
|
|
||||||
vm_name: awx
|
|
||||||
target_remote: oc-node-02
|
|
||||||
ip: 192.168.102.161/22
|
|
||||||
gateway: 192.168.100.1
|
|
||||||
dns: 192.168.100.1
|
|
||||||
cpu: 4
|
|
||||||
memory: 8GiB
|
|
||||||
disk: 40GiB
|
|
||||||
git_repo: ssh://git@192.168.1.200:2222/maarten/incus-contrib.git
|
|
||||||
git_branch: master
|
|
||||||
aether_url: https://192.168.102.160:8443
|
|
||||||
aether_cluster_id: 52
|
|
||||||
|
|
@ -1,26 +0,0 @@
|
||||||
# haproxy.yaml - HAProxy load balancing config for Beelink lab
|
|
||||||
#
|
|
||||||
# Deploys 2 HAProxy containers on the OVN overlay (net-prod, 10.10.10.0/24)
|
|
||||||
# with a VIP on the UPLINK external range (192.168.103.200).
|
|
||||||
#
|
|
||||||
# Architecture:
|
|
||||||
# ffsdn-haproxy-52-01 (10.10.10.50) - HAProxy instance 1 on net-prod
|
|
||||||
# ffsdn-haproxy-52-02 (10.10.10.51) - HAProxy instance 2 on net-prod
|
|
||||||
# nginx-lb-01/02/03 (10.10.10.60-62) - nginx test backends on net-prod
|
|
||||||
# VIP: 192.168.103.200:80 - OVN LB for external access
|
|
||||||
#
|
|
||||||
# Usage:
|
|
||||||
# source envs/beelink/env
|
|
||||||
# bin/deploy-haproxy --config envs/beelink/haproxy.yaml --deploy
|
|
||||||
|
|
||||||
haproxy:
|
|
||||||
cluster_remote: oc-node-01
|
|
||||||
cluster_id: 52
|
|
||||||
ovn_network: net-prod
|
|
||||||
vip: 192.168.103.200
|
|
||||||
haproxy_01_ip: 10.10.10.50
|
|
||||||
haproxy_02_ip: 10.10.10.51
|
|
||||||
backend_ips: 10.10.10.60,10.10.10.61,10.10.10.62
|
|
||||||
service_name: web-test
|
|
||||||
image_version: "1.0.0"
|
|
||||||
aether_url: https://192.168.102.160:8443
|
|
||||||
|
|
@ -1,25 +0,0 @@
|
||||||
# observability.yaml - Monitoring stack config for Beelink lab
|
|
||||||
#
|
|
||||||
# Deploys Prometheus, Grafana, Loki, and Promtail in a monitoring container,
|
|
||||||
# plus node-exporter containers on each cluster node.
|
|
||||||
#
|
|
||||||
# Architecture:
|
|
||||||
# monitoring (10.10.10.70) - Prometheus + Grafana + Loki
|
|
||||||
# node-exp-01/02/03 - node-exporter on each cluster node
|
|
||||||
# Forward VIP: 192.168.103.201 - OVN LB for external access
|
|
||||||
#
|
|
||||||
# Usage:
|
|
||||||
# source envs/beelink/env
|
|
||||||
# bin/deploy-observability --config envs/beelink/observability.yaml --deploy
|
|
||||||
|
|
||||||
observability:
|
|
||||||
cluster_remote: oc-node-01
|
|
||||||
ovn_network: net-prod
|
|
||||||
monitoring_ip: 10.10.10.70
|
|
||||||
monitoring_target: oc-node-02
|
|
||||||
forward_vip: 192.168.103.201
|
|
||||||
incus_nodes: 192.168.102.140,192.168.102.141,192.168.102.142
|
|
||||||
node_exp_targets: oc-node-01,oc-node-02,oc-node-03
|
|
||||||
node_exp_ips: 10.10.10.71,10.10.10.72,10.10.10.73
|
|
||||||
haproxy_ips: 10.10.10.50,10.10.10.51
|
|
||||||
haproxy_cluster_id: 52
|
|
||||||
|
|
@ -1,25 +0,0 @@
|
||||||
# awx.yaml - AWX deployment config for Hetzner lab
|
|
||||||
#
|
|
||||||
# Deploys a Debian 12 VM with K3s + AWX Operator on hz-node-03.
|
|
||||||
# VM uses macvlan parent=mgmt for direct access on the 10.10.0.0/24 network.
|
|
||||||
#
|
|
||||||
# Architecture:
|
|
||||||
# awx (10.10.0.122) - K3s + AWX Operator + AWX instance
|
|
||||||
# AWX exposed on NodePort 30080 (HTTP)
|
|
||||||
#
|
|
||||||
# Usage:
|
|
||||||
# source envs/hetzner/env
|
|
||||||
# bin/deploy-awx --config envs/hetzner/awx.yaml --deploy
|
|
||||||
|
|
||||||
awx:
|
|
||||||
vm_name: awx
|
|
||||||
target_remote: hz-cluster
|
|
||||||
target_node: hz-node-03
|
|
||||||
ip: 10.10.0.122/24
|
|
||||||
gateway: 10.10.0.1
|
|
||||||
dns: 10.10.0.1
|
|
||||||
cpu: 4
|
|
||||||
memory: 8GiB
|
|
||||||
disk: 40GiB
|
|
||||||
aether_url: https://10.10.0.120:8443
|
|
||||||
aether_cluster_id: 1
|
|
||||||
|
|
@ -1,26 +0,0 @@
|
||||||
# haproxy.yaml - HAProxy load balancing config for Hetzner lab
|
|
||||||
#
|
|
||||||
# Deploys 2 HAProxy containers on the OVN overlay (net-prod, 10.10.10.0/24)
|
|
||||||
# with a VIP on the UPLINK external range (10.10.0.200-210).
|
|
||||||
#
|
|
||||||
# Architecture:
|
|
||||||
# ffsdn-haproxy-1-01 (10.10.10.50) - HAProxy instance 1 on net-prod
|
|
||||||
# ffsdn-haproxy-1-02 (10.10.10.51) - HAProxy instance 2 on net-prod
|
|
||||||
# nginx-lb-01/02/03 (10.10.10.60-62) - nginx test backends on net-prod
|
|
||||||
# VIP: 10.10.0.200:80 - OVN external IP for external access
|
|
||||||
#
|
|
||||||
# Usage:
|
|
||||||
# source envs/hetzner/env
|
|
||||||
# bin/deploy-haproxy --config envs/hetzner/haproxy.yaml --deploy
|
|
||||||
|
|
||||||
haproxy:
|
|
||||||
cluster_remote: hz-cluster
|
|
||||||
cluster_id: 1
|
|
||||||
ovn_network: net-prod
|
|
||||||
vip: 10.10.0.200
|
|
||||||
haproxy_01_ip: 10.10.10.50
|
|
||||||
haproxy_02_ip: 10.10.10.51
|
|
||||||
backend_ips: 10.10.10.60,10.10.10.61,10.10.10.62
|
|
||||||
service_name: web-test
|
|
||||||
image_version: "1.0.0"
|
|
||||||
aether_url: https://10.10.0.120:8443
|
|
||||||
|
|
@ -1,25 +0,0 @@
|
||||||
# observability.yaml - Monitoring stack config for Hetzner lab
|
|
||||||
#
|
|
||||||
# Deploys Prometheus, Grafana, Loki, and Promtail in a monitoring container,
|
|
||||||
# plus node-exporter containers on each cluster node.
|
|
||||||
#
|
|
||||||
# Architecture:
|
|
||||||
# monitoring (10.10.10.70) - Prometheus + Grafana + Loki
|
|
||||||
# node-exp-01/02/03 - node-exporter on each cluster node
|
|
||||||
# Forward VIP: 10.10.0.201 - OVN LB for external access
|
|
||||||
#
|
|
||||||
# Usage:
|
|
||||||
# source envs/hetzner/env
|
|
||||||
# bin/deploy-observability --config envs/hetzner/observability.yaml --deploy
|
|
||||||
|
|
||||||
observability:
|
|
||||||
cluster_remote: hz-cluster
|
|
||||||
ovn_network: net-prod
|
|
||||||
monitoring_ip: 10.10.10.70
|
|
||||||
monitoring_target: hz-node-02
|
|
||||||
forward_vip: 10.10.0.201
|
|
||||||
incus_nodes: 10.10.0.111,10.10.0.112,10.10.0.113
|
|
||||||
node_exp_targets: hz-node-01,hz-node-02,hz-node-03
|
|
||||||
node_exp_ips: 10.10.10.71,10.10.10.72,10.10.10.73
|
|
||||||
haproxy_ips: 10.10.10.50,10.10.10.51
|
|
||||||
haproxy_cluster_id: 1
|
|
||||||
|
|
@ -1,12 +0,0 @@
|
||||||
host: 10.10.0.1
|
|
||||||
method: api
|
|
||||||
api_token_id: automation@pve!deploy
|
|
||||||
api_token_secret: f7ed66bc-b1c9-4696-b4a9-c64f158c2a51
|
|
||||||
node: aether-lab
|
|
||||||
storage: local-zfs
|
|
||||||
iso_storage: local
|
|
||||||
bridge: vmbr1
|
|
||||||
gateway: 10.10.0.1
|
|
||||||
dns: 10.10.0.1 # Proxmox host serves DNS for VMs (dnsmasq on vmbr1)
|
|
||||||
pool: IncusLab
|
|
||||||
ssh_user: root
|
|
||||||
|
|
@ -1,907 +0,0 @@
|
||||||
# Hetzner IncusOS Lab — End-to-End Guide
|
|
||||||
|
|
||||||
Full deployment guide for a production-quality IncusOS lab on a Hetzner
|
|
||||||
dedicated server: 1 Operations Center + 3-node Incus cluster with OVN
|
|
||||||
networking, HAProxy load balancing, AWX automation, and observability.
|
|
||||||
|
|
||||||
**Estimated time**: 3-4 hours for a clean run.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Overview
|
|
||||||
|
|
||||||
```
|
|
||||||
Hetzner dedicated (256 GB RAM, 32+ cores)
|
|
||||||
└─ Proxmox VE 9
|
|
||||||
├─ hz-oc (VMID 910) — Operations Center 0.4.x
|
|
||||||
├─ hz-node-01 (VMID 911) — Incus cluster init + OVN control plane
|
|
||||||
├─ hz-node-02 (VMID 912) — Incus cluster member
|
|
||||||
└─ hz-node-03 (VMID 913) — Incus cluster member
|
|
||||||
└─ (OVN overlay 10.10.10.0/24, VIP 10.10.0.200+)
|
|
||||||
├─ ffsdn-haproxy-52-01/02 — HAProxy HA pair
|
|
||||||
├─ nginx-lb-01/02/03 — Test backends
|
|
||||||
├─ monitoring — Prometheus + Grafana
|
|
||||||
├─ node-exp-01/02/03 — Node exporters
|
|
||||||
└─ awx — AWX Operator on K3s
|
|
||||||
```
|
|
||||||
|
|
||||||
**Network topology:**
|
|
||||||
- vmbr0: Public interface (SSH + WireGuard only)
|
|
||||||
- vmbr1: Private bridge 10.10.0.0/24 (VMs + NAT)
|
|
||||||
- wg0: WireGuard tunnel 10.10.99.0/24 (workstation access)
|
|
||||||
- OVN overlay: 10.10.10.0/24 (containers on net-prod)
|
|
||||||
- OVN external IPs: 10.10.0.200+ (VIPs exposed on vmbr1)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prerequisites
|
|
||||||
|
|
||||||
- Hetzner dedicated server with Proxmox VE 9 installed and configured
|
|
||||||
(see [hetzner-setup.md](hetzner-setup.md) + `proxmox-setup`)
|
|
||||||
- WireGuard tunnel active (10.10.99.x → 10.10.0.x accessible)
|
|
||||||
- This repository cloned locally
|
|
||||||
- `env` file populated (see below)
|
|
||||||
- `envs/hetzner/proxmox.yaml` filled in
|
|
||||||
- Incus client installed on workstation
|
|
||||||
|
|
||||||
### env file
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# incusos-contrib/env
|
|
||||||
export PROXMOX_TOKEN_SECRET="<token from proxmox-setup>"
|
|
||||||
export PROXMOX_ROOT_PASSWORD="<proxmox root password>"
|
|
||||||
export AETHER_ADMIN_PASSWORD="admin1234"
|
|
||||||
```
|
|
||||||
|
|
||||||
### proxmox.yaml
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cp envs/hetzner/proxmox.yaml.example \
|
|
||||||
envs/hetzner/proxmox.yaml
|
|
||||||
# Fill in: host, node name, token_id, storage
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 1. Deploy IncusOS VMs
|
|
||||||
|
|
||||||
```bash
|
|
||||||
source env
|
|
||||||
|
|
||||||
bin/incusos-proxmox \
|
|
||||||
--proxmox envs/hetzner/proxmox.yaml \
|
|
||||||
envs/hetzner/lab-production.yaml
|
|
||||||
```
|
|
||||||
|
|
||||||
**What gets deployed:**
|
|
||||||
|
|
||||||
| VMID | Name | IP | RAM | Cores | Disk |
|
|
||||||
|------|-----------|---------------|-------|-------|------|
|
|
||||||
| 910 | hz-oc | 10.10.0.110 | 8 GB | 4 | 100G |
|
|
||||||
| 911 | hz-node-01| 10.10.0.111 | 80 GB | 16 | 150G |
|
|
||||||
| 912 | hz-node-02| 10.10.0.112 | 80 GB | 16 | 100G |
|
|
||||||
| 913 | hz-node-03| 10.10.0.113 | 80 GB | 16 | 100G |
|
|
||||||
|
|
||||||
**Boot takes 3-5 minutes** per VM (sysext download from internet). Do not
|
|
||||||
interrupt. Monitor progress:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Screenshot VM console (requires PROXMOX_ROOT_PASSWORD in env)
|
|
||||||
source env
|
|
||||||
bin/helpers/proxmox-screenshot 910 # hz-oc
|
|
||||||
bin/helpers/proxmox-screenshot 911 # hz-node-01
|
|
||||||
```
|
|
||||||
|
|
||||||
Wait until all VMs are reachable:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
for ip in 10.10.0.110 10.10.0.111 10.10.0.112 10.10.0.113; do
|
|
||||||
echo -n "$ip: "
|
|
||||||
ping -c1 -W2 $ip &>/dev/null && echo "UP" || echo "DOWN"
|
|
||||||
done
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 2. Add Incus remotes
|
|
||||||
|
|
||||||
Add the cluster remote pointing at hz-node-01 (the init node):
|
|
||||||
|
|
||||||
```bash
|
|
||||||
incus remote add hz-cluster https://10.10.0.111:8443 \
|
|
||||||
--accept-certificate --auth-type tls
|
|
||||||
```
|
|
||||||
|
|
||||||
> **Note**: The `incus remote add` command prompts for a trust token.
|
|
||||||
> Obtain the token from OC after cluster formation (section 3), or
|
|
||||||
> if using a pre-formed cluster, generate one with:
|
|
||||||
> `incus config trust add --name workstation`
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 3. Cluster formation via Operations Center
|
|
||||||
|
|
||||||
Open the Operations Center UI:
|
|
||||||
|
|
||||||
```
|
|
||||||
https://10.10.0.110:8443
|
|
||||||
Username: admin
|
|
||||||
Password: admin (first login, then set a new password)
|
|
||||||
```
|
|
||||||
|
|
||||||
### 3.1 Add Incus nodes
|
|
||||||
|
|
||||||
In OC: **Servers → Add Server** for each node:
|
|
||||||
|
|
||||||
| Field | hz-node-01 | hz-node-02 | hz-node-03 |
|
|
||||||
|---------|------------------|------------------|------------------|
|
|
||||||
| Address | 10.10.0.111:8443 | 10.10.0.112:8443 | 10.10.0.113:8443 |
|
|
||||||
| Trust | via token | via token | via token |
|
|
||||||
|
|
||||||
Each node's trust token is displayed on the IncusOS console on first boot.
|
|
||||||
|
|
||||||
### 3.2 Form the cluster
|
|
||||||
|
|
||||||
In OC: **Clusters → Create Cluster**
|
|
||||||
|
|
||||||
- Name: `hz-cluster`
|
|
||||||
- Init node: `hz-node-01`
|
|
||||||
- Members: `hz-node-02`, `hz-node-03`
|
|
||||||
- App seed: `{}` (empty JSON for manual post-formation setup)
|
|
||||||
|
|
||||||
OC forms the cluster automatically. Wait for all 3 nodes to show **Ready**.
|
|
||||||
|
|
||||||
Cluster ID is assigned by OC — note it (e.g. `52`). You'll need it for
|
|
||||||
HAProxy and AWX configuration.
|
|
||||||
|
|
||||||
### 3.3 Verify cluster
|
|
||||||
|
|
||||||
```bash
|
|
||||||
incus cluster list hz-cluster:
|
|
||||||
# Should show 3 members: hz-node-01, hz-node-02, hz-node-03
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 4. OVN networking
|
|
||||||
|
|
||||||
### 4.1 Create the OVN uplink network
|
|
||||||
|
|
||||||
The UPLINK network uses the IncusOS management NIC (`mgmt`) — not
|
|
||||||
`eth0` or `ens18`. This is specific to IncusOS.
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Create UPLINK on each node
|
|
||||||
incus network create hz-cluster:UPLINK --type=physical \
|
|
||||||
--target hz-node-01 \
|
|
||||||
config parent=mgmt
|
|
||||||
|
|
||||||
incus network create hz-cluster:UPLINK --type=physical \
|
|
||||||
--target hz-node-02 \
|
|
||||||
config parent=mgmt
|
|
||||||
|
|
||||||
incus network create hz-cluster:UPLINK --type=physical \
|
|
||||||
--target hz-node-03 \
|
|
||||||
config parent=mgmt
|
|
||||||
|
|
||||||
# Finalize UPLINK on cluster level
|
|
||||||
incus network create hz-cluster:UPLINK \
|
|
||||||
config ipv4.address=none \
|
|
||||||
config ipv6.address=none
|
|
||||||
```
|
|
||||||
|
|
||||||
### 4.2 Deploy OVN central
|
|
||||||
|
|
||||||
OVN central runs as a privileged container on hz-node-01:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
incus launch images:debian/12 hz-cluster:ovn-central \
|
|
||||||
--vm=false \
|
|
||||||
--target hz-node-01 \
|
|
||||||
-c security.privileged=true \
|
|
||||||
-d root,size=20GiB
|
|
||||||
|
|
||||||
# Wait for agent
|
|
||||||
sleep 15
|
|
||||||
|
|
||||||
# Install ovn-central
|
|
||||||
incus exec hz-cluster:ovn-central -- bash -c "
|
|
||||||
apt-get update -qq
|
|
||||||
apt-get install -y -qq ovn-central
|
|
||||||
systemctl enable ovn-northd ovn-ovsdb-server-nb ovn-ovsdb-server-sb
|
|
||||||
systemctl start ovn-northd ovn-ovsdb-server-nb ovn-ovsdb-server-sb
|
|
||||||
"
|
|
||||||
```
|
|
||||||
|
|
||||||
### 4.3 Expose OVN NB/SB ports to cluster
|
|
||||||
|
|
||||||
Create proxy devices so cluster nodes can reach OVN:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Get ovn-central IP
|
|
||||||
OVN_IP=$(incus list hz-cluster:ovn-central --format csv -c 4 | head -1 | awk '{print $1}')
|
|
||||||
|
|
||||||
incus config device add hz-cluster:ovn-central ovn-nb proxy \
|
|
||||||
listen="tcp:${OVN_IP}:6641" connect="tcp:127.0.0.1:6641" \
|
|
||||||
bind=host
|
|
||||||
incus config device add hz-cluster:ovn-central ovn-sb proxy \
|
|
||||||
listen="tcp:${OVN_IP}:6642" connect="tcp:127.0.0.1:6642" \
|
|
||||||
bind=host
|
|
||||||
```
|
|
||||||
|
|
||||||
### 4.4 Create OVN overlay network
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Use hz-node-01 IP as OVN NB/SB server
|
|
||||||
NODE01_IP="10.10.0.111"
|
|
||||||
|
|
||||||
incus network create hz-cluster:net-prod \
|
|
||||||
--type=ovn \
|
|
||||||
config network=UPLINK \
|
|
||||||
config ipv4.address=10.10.10.1/24 \
|
|
||||||
config ipv4.nat=true \
|
|
||||||
config ipv6.address=none \
|
|
||||||
config ovn.northbound_connection="tcp:${NODE01_IP}:6641" \
|
|
||||||
config ovn.southbound_connection="tcp:${NODE01_IP}:6642"
|
|
||||||
```
|
|
||||||
|
|
||||||
Wait for the network to reach **Created** state:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
incus network info hz-cluster:net-prod
|
|
||||||
```
|
|
||||||
|
|
||||||
### 4.5 Configure external IP range (OVN VIPs)
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Reserve 10.10.0.200-210 for OVN external IPs (VIPs)
|
|
||||||
# These are allocated on the Proxmox host's vmbr1 bridge
|
|
||||||
# No additional configuration needed — OVN will ARP for these IPs
|
|
||||||
```
|
|
||||||
|
|
||||||
Verify:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
incus network list hz-cluster:
|
|
||||||
# net-prod should show type: ovn, state: Created
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 5. Deploy Aether
|
|
||||||
|
|
||||||
### 5.1 Deploy the Aether VM
|
|
||||||
|
|
||||||
Use the macvlan method to give Aether a direct vmbr1 IP:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
incus launch images:ubuntu/22.04 hz-cluster:aether \
|
|
||||||
--vm \
|
|
||||||
--target hz-node-01 \
|
|
||||||
-d root,size=40GiB \
|
|
||||||
-c limits.cpu=4 \
|
|
||||||
-c limits.memory=8GiB \
|
|
||||||
-d eth0,type=nic,nictype=macvlan,parent=mgmt,ipv4.address=10.10.0.120
|
|
||||||
```
|
|
||||||
|
|
||||||
> **Alternative**: use Aether golden image if already built on the cluster.
|
|
||||||
> Check `incus image list hz-cluster: | grep aether`.
|
|
||||||
|
|
||||||
Wait for the VM, then SSH in:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
ssh ubuntu@10.10.0.120
|
|
||||||
```
|
|
||||||
|
|
||||||
Follow the [Aether installation guide](../notes/aether-guide.md) for the
|
|
||||||
full Aether setup. Aether is accessible at `https://10.10.0.120:8443`.
|
|
||||||
|
|
||||||
### 5.2 Configure Aether cluster
|
|
||||||
|
|
||||||
In Aether UI (`https://10.10.0.120:8443`):
|
|
||||||
|
|
||||||
1. **Settings → Clusters → Add Cluster**
|
|
||||||
- Name: hz-cluster
|
|
||||||
- Endpoint: `https://10.10.0.111:8443`
|
|
||||||
- Authenticate (generates a trust token, add it to the cluster)
|
|
||||||
|
|
||||||
2. **Settings → Operations Centers → Add OC**
|
|
||||||
- URL: `https://10.10.0.110:8443`
|
|
||||||
|
|
||||||
3. Verify hz-cluster instances sync in Aether dashboard.
|
|
||||||
|
|
||||||
Aether default credentials: `admin` / `admin` (change on first login).
|
|
||||||
|
|
||||||
For Aether password, set in `env`:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
export AETHER_ADMIN_PASSWORD="admin1234"
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 6. HAProxy load balancing
|
|
||||||
|
|
||||||
### 6.1 Create config file
|
|
||||||
|
|
||||||
Save as `envs/hetzner/hz-haproxy.yaml` (or `/tmp/hz-deploy.yaml`):
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
haproxy:
|
|
||||||
cluster_remote: hz-cluster
|
|
||||||
cluster_id: 52 # from OC cluster listing
|
|
||||||
ovn_network: net-prod
|
|
||||||
vip: 10.10.0.200
|
|
||||||
haproxy_01_ip: 10.10.10.50
|
|
||||||
haproxy_02_ip: 10.10.10.51
|
|
||||||
backend_ips: 10.10.10.60,10.10.10.61,10.10.10.62
|
|
||||||
service_name: web-test
|
|
||||||
image_version: "1.0.0"
|
|
||||||
aether_url: https://10.10.0.120:8443
|
|
||||||
```
|
|
||||||
|
|
||||||
> **Cluster ID**: Find it via OC API or in Aether UI under Clusters.
|
|
||||||
> It's the numeric ID assigned when the cluster was created.
|
|
||||||
|
|
||||||
### 6.2 Build HAProxy image
|
|
||||||
|
|
||||||
Before deploying infrastructure, build and push the HAProxy image:
|
|
||||||
|
|
||||||
In Aether UI → **HAProxy → Images → Build**:
|
|
||||||
- Version: `1.0.0`
|
|
||||||
- Cluster: `hz-cluster`
|
|
||||||
- Click **Build** and wait 5-10 minutes
|
|
||||||
|
|
||||||
After build, push to hz-cluster:
|
|
||||||
- Click **Push** → select `hz-cluster`
|
|
||||||
- Click **Set Current**
|
|
||||||
|
|
||||||
Alternatively, use the script (requires the UI steps above for first build):
|
|
||||||
|
|
||||||
```bash
|
|
||||||
source env
|
|
||||||
bin/deploy-haproxy -c /tmp/hz-deploy.yaml --deploy --skip-image
|
|
||||||
```
|
|
||||||
|
|
||||||
### 6.3 Full deploy
|
|
||||||
|
|
||||||
```bash
|
|
||||||
source env
|
|
||||||
bin/deploy-haproxy -c /tmp/hz-deploy.yaml --deploy
|
|
||||||
```
|
|
||||||
|
|
||||||
This:
|
|
||||||
1. Launches 3 nginx backend containers (`nginx-lb-01/02/03`) on `net-prod`
|
|
||||||
2. Deploys HAProxy infrastructure via Aether (2 containers + OVN VIP)
|
|
||||||
3. Creates the `web-test` service (HTTP, round-robin, health checks)
|
|
||||||
|
|
||||||
On subsequent runs (if backends already exist):
|
|
||||||
|
|
||||||
```bash
|
|
||||||
bin/deploy-haproxy -c /tmp/hz-deploy.yaml --deploy \
|
|
||||||
--skip-backends --skip-image
|
|
||||||
```
|
|
||||||
|
|
||||||
### 6.4 Verify
|
|
||||||
|
|
||||||
```bash
|
|
||||||
curl http://10.10.0.200/
|
|
||||||
# Should show "Backend: nginx-lb-0x"
|
|
||||||
|
|
||||||
# Test distribution
|
|
||||||
for i in $(seq 1 6); do curl -s http://10.10.0.200/ | grep 'Backend:'; done
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 7. AWX automation
|
|
||||||
|
|
||||||
### 7.1 Create config file
|
|
||||||
|
|
||||||
Save as `envs/hetzner/awx.yaml`:
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
awx:
|
|
||||||
vm_name: awx
|
|
||||||
target_remote: hz-cluster
|
|
||||||
target_node: hz-node-03 # specific node within cluster
|
|
||||||
ip: 10.10.0.122/24
|
|
||||||
gateway: 10.10.0.1
|
|
||||||
dns: 10.10.0.1
|
|
||||||
cpu: 4
|
|
||||||
memory: 8GiB
|
|
||||||
disk: 40GiB
|
|
||||||
aether_url: https://10.10.0.120:8443
|
|
||||||
aether_cluster_id: 52
|
|
||||||
```
|
|
||||||
|
|
||||||
> **target_node vs target_remote**: `target_remote` is the Incus remote
|
|
||||||
> name (e.g. `hz-cluster`), while `target_node` is the specific cluster
|
|
||||||
> member to deploy on (e.g. `hz-node-03`). Both are required for
|
|
||||||
> cluster deployments.
|
|
||||||
|
|
||||||
### 7.2 Deploy
|
|
||||||
|
|
||||||
```bash
|
|
||||||
source env
|
|
||||||
bin/deploy-awx -c envs/hetzner/awx.yaml --deploy
|
|
||||||
```
|
|
||||||
|
|
||||||
This deploys a Debian 12 VM with K3s + AWX Operator. Takes ~20 minutes.
|
|
||||||
AWX will be at `http://10.10.0.122:30080`.
|
|
||||||
|
|
||||||
On partial failure (e.g. network issue mid-deploy), resume:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
bin/deploy-awx -c envs/hetzner/awx.yaml --deploy --resume
|
|
||||||
# Skips VM creation and K3s install phases, jumps to AWX Operator phase
|
|
||||||
```
|
|
||||||
|
|
||||||
If you get `permission denied` on temp files from a previous aborted run:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
incus exec hz-cluster:awx -- rm -f \
|
|
||||||
/tmp/awx-operator-kustomization.yaml \
|
|
||||||
/tmp/awx-kustomization.yaml \
|
|
||||||
/tmp/awx-instance.yaml
|
|
||||||
```
|
|
||||||
|
|
||||||
### 7.3 Configure AWX
|
|
||||||
|
|
||||||
> **Wait for the awx-task pod before running --configure.** The AWX web
|
|
||||||
> deployment becomes ready first (~3 min), but `--configure` needs the
|
|
||||||
> task pod. Check it:
|
|
||||||
> ```bash
|
|
||||||
> incus exec hz-cluster:awx -- kubectl -n awx wait \
|
|
||||||
> --for=condition=Ready pod -l app.kubernetes.io/name=awx-task \
|
|
||||||
> --timeout=300s
|
|
||||||
> ```
|
|
||||||
|
|
||||||
```bash
|
|
||||||
source env
|
|
||||||
bin/deploy-awx -c envs/hetzner/awx.yaml --configure
|
|
||||||
```
|
|
||||||
|
|
||||||
This creates:
|
|
||||||
- Project: `incus-contrib` (linked to your git repo)
|
|
||||||
- Inventory: `incus-lab`
|
|
||||||
- Templates: `post-deploy` (ID 9), `decommission` (ID 10), and lifecycle templates
|
|
||||||
|
|
||||||
Get AWX admin password:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
bin/deploy-awx -c /tmp/hz-awx.yaml --status
|
|
||||||
```
|
|
||||||
|
|
||||||
### 7.4 Join Aether
|
|
||||||
|
|
||||||
Generate an AWX API token for Aether:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
source env
|
|
||||||
bin/deploy-awx -c envs/hetzner/awx.yaml --join-aether
|
|
||||||
# Prints the PAT token and template IDs
|
|
||||||
```
|
|
||||||
|
|
||||||
In Aether UI → **Ansible Automation** → **+ Add Endpoint**:
|
|
||||||
- Name: `lab-awx`
|
|
||||||
- URL: `http://10.10.0.122:30080`
|
|
||||||
- Token: (paste token from `--join-aether` output)
|
|
||||||
- Verify SSL: unchecked (HTTP endpoint)
|
|
||||||
|
|
||||||
Click **Test Connection** (should show `Connected! AWX version: 24.x.x`) then **Save Endpoint**.
|
|
||||||
|
|
||||||
### 7.5 Bind AWX to cluster
|
|
||||||
|
|
||||||
**Note**: In Aether ≤ v6.4.202, the `PUT /api/clusters/{id}/awx-config`
|
|
||||||
endpoint returns `400 "Invalid cluster ID"` due to a bug.
|
|
||||||
|
|
||||||
**Workaround** (direct DB update):
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# First confirm the AWX endpoint ID (assigned by the database):
|
|
||||||
incus exec hz-cluster:aether -- bash -c \
|
|
||||||
"su - postgres -c \"psql -d incusovnsdnc -c 'SELECT id, name, awx_url FROM awx_endpoints;'\""
|
|
||||||
|
|
||||||
# Then apply the binding (adjust awx_endpoint_id if yours differs from 2):
|
|
||||||
incus exec hz-cluster:aether -- bash -c "su - postgres -c \"psql -d incusovnsdnc -c \\\"
|
|
||||||
UPDATE incus_ovn_clusters
|
|
||||||
SET awx_endpoint_id=2,
|
|
||||||
awx_post_deploy_template_id=9,
|
|
||||||
awx_decommission_template_id=10,
|
|
||||||
awx_job_timeout_seconds=600
|
|
||||||
WHERE cluster_id=52;
|
|
||||||
\\\"\""
|
|
||||||
```
|
|
||||||
|
|
||||||
> **Note**: The database is PostgreSQL (`incusovnsdnc`), accessed as the
|
|
||||||
> `postgres` system user. The endpoint ID is auto-assigned — always check
|
|
||||||
> it before applying the fix. Template IDs 9 and 10 are set by `--configure`.
|
|
||||||
|
|
||||||
> **Check future Aether versions**: Run `curl -sk https://10.10.0.120:8443/api/swagger.yaml | grep awx-config`
|
|
||||||
> to see if the endpoint has been fixed. If it returns a valid schema,
|
|
||||||
> use the API instead of the DB workaround.
|
|
||||||
|
|
||||||
Verify in Aether UI: **Clusters → hz-cluster → Settings** should show
|
|
||||||
the AWX endpoint, post-deploy, and decommission templates.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 8. Observability stack
|
|
||||||
|
|
||||||
### 8.1 Deploy
|
|
||||||
|
|
||||||
```bash
|
|
||||||
source env
|
|
||||||
bin/deploy-observability \
|
|
||||||
--remote hz-cluster \
|
|
||||||
--monitoring-target hz-node-01 \
|
|
||||||
--monitoring-ip 10.10.10.70 \
|
|
||||||
--forward-vip 10.10.0.201 \
|
|
||||||
--deploy
|
|
||||||
```
|
|
||||||
|
|
||||||
> **Note**: Do NOT pass `--oc-server` for the Hetzner setup. The
|
|
||||||
> Operations Center (hz-oc) runs OC, not standard Incus — its
|
|
||||||
> `/1.0/instances` endpoint returns 404 and will fail the node
|
|
||||||
> discovery phase.
|
|
||||||
|
|
||||||
> **Grafana install may fail if net-prod containers lack internet.**
|
|
||||||
> OVN SNAT (outbound from 10.10.10.0/24 via 10.10.0.200) works briefly
|
|
||||||
> after container launch but can become unreliable. If phase 4 fails:
|
|
||||||
>
|
|
||||||
> ```bash
|
|
||||||
> # Download Grafana locally and push it in
|
|
||||||
> curl -L -o /tmp/grafana.deb https://dl.grafana.com/oss/release/grafana_12.4.0_amd64.deb
|
|
||||||
> incus file push /tmp/grafana.deb hz-cluster:monitoring/tmp/grafana.deb
|
|
||||||
> incus exec hz-cluster:monitoring -- bash -c "
|
|
||||||
> export DEBIAN_FRONTEND=noninteractive
|
|
||||||
> dpkg -i /tmp/grafana.deb; apt-get -f install -y
|
|
||||||
> mkdir -p /etc/grafana/provisioning/datasources
|
|
||||||
> cat > /etc/grafana/provisioning/datasources/observability.yaml << 'EOF'
|
|
||||||
> apiVersion: 1
|
|
||||||
> datasources:
|
|
||||||
> - name: Prometheus
|
|
||||||
> type: prometheus
|
|
||||||
> uid: prometheus
|
|
||||||
> access: proxy
|
|
||||||
> url: http://localhost:9090
|
|
||||||
> isDefault: true
|
|
||||||
> - name: Loki
|
|
||||||
> type: loki
|
|
||||||
> uid: loki
|
|
||||||
> access: proxy
|
|
||||||
> url: http://localhost:3100
|
|
||||||
> EOF
|
|
||||||
> systemctl daemon-reload
|
|
||||||
> systemctl enable grafana-server
|
|
||||||
> systemctl start grafana-server
|
|
||||||
> "
|
|
||||||
> ```
|
|
||||||
>
|
|
||||||
> Then continue with `--resume --deploy` (skips phases 1-5, runs ACLs,
|
|
||||||
> network forward, and dashboards).
|
|
||||||
|
|
||||||
> **Node-exporter containers are skipped by `--resume`.** If internet
|
|
||||||
> was unavailable during phase 5, deploy them manually:
|
|
||||||
>
|
|
||||||
> ```bash
|
|
||||||
> # Download node_exporter binary locally
|
|
||||||
> curl -sL -o /tmp/ne.tar.gz \
|
|
||||||
> https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz
|
|
||||||
> tar xzf /tmp/ne.tar.gz -C /tmp node_exporter-1.8.2.linux-amd64/node_exporter
|
|
||||||
>
|
|
||||||
> for i in 1 2 3; do
|
|
||||||
> name="node-exp-0${i}"
|
|
||||||
> ip="10.10.10.7${i}"
|
|
||||||
> node="hz-node-0${i}"
|
|
||||||
>
|
|
||||||
> incus launch images:alpine/3.20 hz-cluster:${name} \
|
|
||||||
> -n net-prod --target ${node} \
|
|
||||||
> -c limits.memory=128MiB -c security.privileged=true
|
|
||||||
> incus config device set hz-cluster:${name} eth0 ipv4.address=${ip}
|
|
||||||
> sleep 5
|
|
||||||
>
|
|
||||||
> for dev in proc sys root; do
|
|
||||||
> src="/$( [[ $dev == root ]] && echo '' || echo $dev)"; src="${src:-/}"
|
|
||||||
> incus config device add hz-cluster:${name} host-${dev} disk \
|
|
||||||
> source=/${dev} path=/host/${dev} readonly=true 2>/dev/null || \
|
|
||||||
> incus config device add hz-cluster:${name} host-${dev} disk \
|
|
||||||
> source=/ path=/host/root readonly=true 2>/dev/null || true
|
|
||||||
> done
|
|
||||||
>
|
|
||||||
> incus file push /tmp/node_exporter-1.8.2.linux-amd64/node_exporter \
|
|
||||||
> hz-cluster:${name}/usr/bin/node_exporter
|
|
||||||
>
|
|
||||||
> incus exec hz-cluster:${name} -- sh -c "
|
|
||||||
> chmod +x /usr/bin/node_exporter
|
|
||||||
> cat > /etc/init.d/node-exporter << 'INIT'
|
|
||||||
> #!/sbin/openrc-run
|
|
||||||
> description=\"Node Exporter\"
|
|
||||||
> command=/usr/bin/node_exporter
|
|
||||||
> command_args=\"--path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/host/root\"
|
|
||||||
> command_background=true
|
|
||||||
> pidfile=/run/node-exporter.pid
|
|
||||||
> INIT
|
|
||||||
> chmod +x /etc/init.d/node-exporter
|
|
||||||
> rc-update add node-exporter default
|
|
||||||
> rc-service node-exporter start
|
|
||||||
>
|
|
||||||
> cat > /etc/network/interfaces << NETCFG
|
|
||||||
> auto lo
|
|
||||||
> iface lo inet loopback
|
|
||||||
> auto eth0
|
|
||||||
> iface eth0 inet static
|
|
||||||
> address ${ip}/24
|
|
||||||
> gateway 10.10.10.1
|
|
||||||
> NETCFG
|
|
||||||
> rc-service networking restart
|
|
||||||
> "
|
|
||||||
>
|
|
||||||
> incus config device set hz-cluster:${name} eth0 \
|
|
||||||
> security.acls=monitoring-allow \
|
|
||||||
> security.acls.default.ingress.action=allow \
|
|
||||||
> security.acls.default.egress.action=allow
|
|
||||||
> done
|
|
||||||
> ```
|
|
||||||
|
|
||||||
On partial failure (phases 6-9 only), resume:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
source env
|
|
||||||
bin/deploy-observability \
|
|
||||||
--remote hz-cluster \
|
|
||||||
--monitoring-target hz-node-01 \
|
|
||||||
--monitoring-ip 10.10.10.70 \
|
|
||||||
--forward-vip 10.10.0.201 \
|
|
||||||
--resume --deploy
|
|
||||||
```
|
|
||||||
|
|
||||||
After deploying, fix the HAProxy prometheus exporter (default stats page
|
|
||||||
doesn't expose `/metrics`):
|
|
||||||
|
|
||||||
```bash
|
|
||||||
for ct in ffsdn-haproxy-52-01 ffsdn-haproxy-52-02; do
|
|
||||||
incus exec hz-cluster:${ct} -- bash -c "
|
|
||||||
sed -i 's/ stats enable/ http-request use-service prometheus-exporter if { path \/metrics }\n stats enable/' /etc/haproxy/haproxy.cfg
|
|
||||||
systemctl reload haproxy
|
|
||||||
"
|
|
||||||
done
|
|
||||||
```
|
|
||||||
|
|
||||||
### 8.2 Access
|
|
||||||
|
|
||||||
| Service | URL | Credentials |
|
|
||||||
|------------|------------------------------|-------------|
|
|
||||||
| Grafana | http://10.10.0.201:3000 | admin/admin |
|
|
||||||
| Prometheus | http://10.10.0.201:9090 | — |
|
|
||||||
|
|
||||||
9 dashboards are pre-loaded: cluster overview, node metrics, HAProxy traffic,
|
|
||||||
storage, networking, capacity planning, and log explorer.
|
|
||||||
|
|
||||||
Verify all Prometheus targets are up:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
curl -s http://10.10.0.201:9090/api/v1/targets | python3 -c "
|
|
||||||
import sys,json
|
|
||||||
for t in json.load(sys.stdin)['data']['activeTargets']:
|
|
||||||
print(t['labels'].get('job','?'), t['health'], t['scrapeUrl'][:50])
|
|
||||||
"
|
|
||||||
# Expected: 2 haproxy, 3 incus, 3 node-exporter, 1 prometheus — all 'up'
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 9. WireGuard access (macOS)
|
|
||||||
|
|
||||||
If you use the macOS WireGuard app from the App Store:
|
|
||||||
|
|
||||||
- The app uses Network Extension (visible as `utun5` in `ifconfig`)
|
|
||||||
- `wg show` always shows empty — this is normal with the App Store app
|
|
||||||
- The app allows only **one active tunnel at a time**
|
|
||||||
|
|
||||||
For a second simultaneous tunnel:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Install wg-quick for macOS
|
|
||||||
brew install wireguard-tools
|
|
||||||
|
|
||||||
# Start the second tunnel alongside the App Store one
|
|
||||||
sudo wg-quick up ~/.config/wireguard/hetzner.conf
|
|
||||||
```
|
|
||||||
|
|
||||||
If the server added a new WireGuard peer (new workstation/client):
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# On the Hetzner Proxmox host (via the existing tunnel or public IP)
|
|
||||||
ssh hetzner-lab "wg set wg0 peer <public-key> allowed-ips 10.10.99.x/32"
|
|
||||||
ssh hetzner-lab "wg-quick save wg0"
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 10. Incus remote management
|
|
||||||
|
|
||||||
### Add remotes
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# OC (Operations Center — not standard Incus)
|
|
||||||
# Note: Must be added manually — incus remote add loops prompting for token
|
|
||||||
# OC does not support the standard trust token flow.
|
|
||||||
# Edit ~/.config/incus/config.yml directly:
|
|
||||||
```
|
|
||||||
|
|
||||||
Add to `~/.config/incus/config.yml`:
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
remotes:
|
|
||||||
oc-server:
|
|
||||||
addr: https://10.10.0.110:8443
|
|
||||||
auth_type: tls
|
|
||||||
protocol: incus
|
|
||||||
public: false
|
|
||||||
hz-cluster:
|
|
||||||
addr: https://10.10.0.111:8443
|
|
||||||
auth_type: tls
|
|
||||||
protocol: incus
|
|
||||||
public: false
|
|
||||||
```
|
|
||||||
|
|
||||||
Copy the OC certificate (accept it with curl first):
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Accept and save OC cert
|
|
||||||
curl -sk https://10.10.0.110:8443 > /dev/null
|
|
||||||
# The cert ends up in your browser cache — use openssl to fetch it
|
|
||||||
openssl s_client -connect 10.10.0.110:8443 </dev/null 2>/dev/null \
|
|
||||||
| openssl x509 > ~/.config/incus/servercerts/oc-server.crt
|
|
||||||
```
|
|
||||||
|
|
||||||
> **Warning**: `oc-server` is OC, not Incus. `incus list oc-server:` will
|
|
||||||
> fail. OC does not expose the standard `/1.0/instances` endpoint.
|
|
||||||
|
|
||||||
### After redeploy (remote cert changed)
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Remove old remote and re-add
|
|
||||||
incus remote remove hz-cluster
|
|
||||||
incus remote add hz-cluster https://10.10.0.111:8443 \
|
|
||||||
--accept-certificate --auth-type tls
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 11. Quick reference
|
|
||||||
|
|
||||||
### Access URLs
|
|
||||||
|
|
||||||
| Service | URL | Credentials |
|
|
||||||
|-----------------|----------------------------------|---------------------|
|
|
||||||
| Operations Center | https://10.10.0.110:8443 | admin / (set on first login) |
|
|
||||||
| Aether | https://10.10.0.120:8443 | admin / admin1234 |
|
|
||||||
| AWX | http://10.10.0.122:30080 | admin / (from deploy-awx --status) |
|
|
||||||
| Grafana | http://10.10.0.201:3000 | admin / admin |
|
|
||||||
| Prometheus | http://10.10.0.201:9090 | — |
|
|
||||||
| HAProxy VIP | http://10.10.0.200/ | — |
|
|
||||||
|
|
||||||
### IP allocation
|
|
||||||
|
|
||||||
| Range | Purpose |
|
|
||||||
|-----------------------|------------------------------|
|
|
||||||
| 10.10.0.1 | Proxmox host (vmbr1 gateway) |
|
|
||||||
| 10.10.0.110-113 | IncusOS VMs (OC + nodes) |
|
|
||||||
| 10.10.0.120 | Aether VM (macvlan) |
|
|
||||||
| 10.10.0.122 | AWX VM (macvlan) |
|
|
||||||
| 10.10.0.200+ | OVN external IPs (VIPs) |
|
|
||||||
| 10.10.10.0/24 | OVN overlay (containers) |
|
|
||||||
| 10.10.10.50-51 | HAProxy containers |
|
|
||||||
| 10.10.10.60-62 | nginx test backends |
|
|
||||||
| 10.10.10.70 | monitoring container |
|
|
||||||
| 10.10.10.71-73 | node-exporter containers |
|
|
||||||
| 10.10.99.0/24 | WireGuard tunnel |
|
|
||||||
| 10.10.99.1 | WireGuard server |
|
|
||||||
| 10.10.99.2 | Your workstation |
|
|
||||||
|
|
||||||
### Cluster IDs
|
|
||||||
|
|
||||||
- hz-cluster (OC): ID 52 (assigned by OC at cluster creation — check
|
|
||||||
your OC UI if different)
|
|
||||||
|
|
||||||
### Useful incus commands
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# List all instances in cluster
|
|
||||||
incus list hz-cluster:
|
|
||||||
|
|
||||||
# List cluster members
|
|
||||||
incus cluster list hz-cluster:
|
|
||||||
|
|
||||||
# List networks
|
|
||||||
incus network list hz-cluster:
|
|
||||||
|
|
||||||
# Exec into a container
|
|
||||||
incus exec hz-cluster:monitoring -- bash
|
|
||||||
|
|
||||||
# Start/stop a container
|
|
||||||
incus start hz-cluster:monitoring
|
|
||||||
incus stop hz-cluster:monitoring --force
|
|
||||||
|
|
||||||
# Check container network
|
|
||||||
incus exec hz-cluster:nginx-lb-01 -- ip a
|
|
||||||
```
|
|
||||||
|
|
||||||
### Teardown
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Force destroy all VMs (no nice cleanup needed for test lab)
|
|
||||||
source env
|
|
||||||
for vmid in 910 911 912 913; do
|
|
||||||
bin/helpers/proxmox-api POST \
|
|
||||||
"/nodes/aether-lab/qemu/${vmid}/status/stop" \
|
|
||||||
'{"forceStop":1}' 2>/dev/null || true
|
|
||||||
sleep 3
|
|
||||||
bin/helpers/proxmox-api DELETE \
|
|
||||||
"/nodes/aether-lab/qemu/${vmid}?purge=1&destroy-unreferenced-disks=1"
|
|
||||||
done
|
|
||||||
|
|
||||||
# Remove stale remotes
|
|
||||||
incus remote remove hz-cluster 2>/dev/null || true
|
|
||||||
incus remote remove oc-server 2>/dev/null || true
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Known issues and workarounds
|
|
||||||
|
|
||||||
### OC remote add loops (incus remote add oc-server fails)
|
|
||||||
|
|
||||||
`incus remote add` enters a trust token prompt loop for OC because OC
|
|
||||||
does not implement the standard Incus trust token flow.
|
|
||||||
|
|
||||||
**Workaround**: Add the remote manually to `~/.config/incus/config.yml`
|
|
||||||
and save the TLS certificate separately.
|
|
||||||
|
|
||||||
### IncusOS management NIC name is `mgmt`, not `eth0`
|
|
||||||
|
|
||||||
All IncusOS instances use `mgmt` as the management NIC name. Use
|
|
||||||
`parent=mgmt` in UPLINK network config, not `parent=eth0` or `parent=ens18`.
|
|
||||||
|
|
||||||
### Aether `PUT /api/clusters/{id}/awx-config` returns 400
|
|
||||||
|
|
||||||
Bug in Aether ≤ v6.4.202. Use the direct DB workaround (section 7.5).
|
|
||||||
|
|
||||||
### deploy-observability: don't use --oc-server with OC
|
|
||||||
|
|
||||||
The `--oc-server` flag tries to list instances via the standard Incus
|
|
||||||
API (`/1.0/instances`), which OC does not implement. Omit the flag.
|
|
||||||
|
|
||||||
### OVN SNAT internet access from net-prod containers is unreliable
|
|
||||||
|
|
||||||
Containers on `net-prod` (10.10.10.0/24) use SNAT via 10.10.0.200 to
|
|
||||||
reach the internet. This works briefly after container launch (via Proxmox
|
|
||||||
NAT masquerade) but can become unreliable as ARP for the VIP IP becomes
|
|
||||||
stale. Connections return `Connection refused` from Fastly CDN.
|
|
||||||
|
|
||||||
**Workaround**: Download packages locally and push them in with
|
|
||||||
`incus file push` instead of relying on apt/apk inside net-prod containers.
|
|
||||||
|
|
||||||
Containers on macvlan mgmt (like AWX, Aether) have reliable internet via
|
|
||||||
Proxmox masquerade and are unaffected.
|
|
||||||
|
|
||||||
### deploy-awx temp file permission errors on resume
|
|
||||||
|
|
||||||
Temp files created as root inside the AWX VM persist between runs.
|
|
||||||
Clean up before resuming:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
incus exec hz-cluster:awx -- rm -f \
|
|
||||||
/tmp/awx-operator-kustomization.yaml \
|
|
||||||
/tmp/awx-kustomization.yaml \
|
|
||||||
/tmp/awx-instance.yaml
|
|
||||||
```
|
|
||||||
|
|
||||||
### aether_url in config files must use key-specific sed
|
|
||||||
|
|
||||||
Config parsing uses `sed 's/^ *aether_url: *//'` (key-specific match).
|
|
||||||
A greedy `sed 's/.*: *//'` would strip the `:8443` port. This is
|
|
||||||
already fixed in all scripts in this repo.
|
|
||||||
|
|
@ -51,12 +51,12 @@ and API tokens for `incusos-proxmox`.
|
||||||
### 4. Deploy IncusOS VMs
|
### 4. Deploy IncusOS VMs
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cp envs/hetzner/proxmox.yaml.example envs/hetzner/proxmox.yaml
|
cp incusos/targets/hetzner/proxmox.yaml.example incusos/targets/hetzner/proxmox.yaml
|
||||||
# Fill in the API token from step 3
|
# Fill in the API token from step 3
|
||||||
|
|
||||||
export PROXMOX_TOKEN_SECRET="<token>"
|
export PROXMOX_TOKEN_SECRET="<token>"
|
||||||
bin/incusos-proxmox --proxmox envs/hetzner/proxmox.yaml \
|
incusos-proxmox --proxmox incusos/targets/hetzner/proxmox.yaml \
|
||||||
envs/hetzner/lab-cluster.yaml
|
incusos/targets/hetzner/lab-cluster.yaml
|
||||||
```
|
```
|
||||||
|
|
||||||
## What's in this directory
|
## What's in this directory
|
||||||
|
|
@ -92,5 +92,5 @@ Internet
|
||||||
|
|
||||||
## Related files
|
## Related files
|
||||||
|
|
||||||
- [`envs/hetzner/`](../) -- Proxmox connection configs and lab definitions
|
- [`incusos/targets/hetzner/`](../incusos/targets/hetzner/) -- Proxmox connection configs and lab definitions
|
||||||
- [`envs/README.md`](../../README.md) -- Multi-target concept explanation
|
- [`incusos/targets/README.md`](../incusos/targets/README.md) -- Multi-target concept explanation
|
||||||
|
|
@ -295,7 +295,7 @@ Each step prompts for confirmation. Use `--yes` to skip prompts, or
|
||||||
`--skip-repos`, `--skip-storage`, etc. to skip individual steps.
|
`--skip-repos`, `--skip-storage`, etc. to skip individual steps.
|
||||||
|
|
||||||
At the end, the script prints:
|
At the end, the script prints:
|
||||||
- A ready-to-use `proxmox.yaml` config for `envs/hetzner/`
|
- A ready-to-use `proxmox.yaml` config for `incusos/targets/hetzner/`
|
||||||
- An `env` file snippet with the API token
|
- An `env` file snippet with the API token
|
||||||
- A WireGuard client config to save on your workstation
|
- A WireGuard client config to save on your workstation
|
||||||
|
|
||||||
|
|
@ -330,15 +330,15 @@ Host hetzner-lab-wg
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Copy and fill in the connection config
|
# Copy and fill in the connection config
|
||||||
cp envs/hetzner/proxmox.yaml.example envs/hetzner/proxmox.yaml
|
cp incusos/targets/hetzner/proxmox.yaml.example incusos/targets/hetzner/proxmox.yaml
|
||||||
|
|
||||||
# Set the API token
|
# Set the API token
|
||||||
export PROXMOX_TOKEN_SECRET="<token from proxmox-setup output>"
|
export PROXMOX_TOKEN_SECRET="<token from proxmox-setup output>"
|
||||||
|
|
||||||
# Dry run -- verify correct bridge, IPs, storage
|
# Dry run -- verify correct bridge, IPs, storage
|
||||||
bin/incusos-proxmox --dry-run \
|
incusos-proxmox --dry-run \
|
||||||
--proxmox envs/hetzner/proxmox.yaml \
|
--proxmox incusos/targets/hetzner/proxmox.yaml \
|
||||||
envs/hetzner/lab-cluster.yaml
|
incusos/targets/hetzner/lab-cluster.yaml
|
||||||
```
|
```
|
||||||
|
|
||||||
Confirm the output shows: `vmbr1` bridge, `10.10.0.x` IPs, `local-zfs`
|
Confirm the output shows: `vmbr1` bridge, `10.10.0.x` IPs, `local-zfs`
|
||||||
|
|
@ -34,6 +34,7 @@ WG_PORT="51820"
|
||||||
SKIP_REPOS=false
|
SKIP_REPOS=false
|
||||||
SKIP_WIREGUARD=false
|
SKIP_WIREGUARD=false
|
||||||
SKIP_FIREWALL=false
|
SKIP_FIREWALL=false
|
||||||
|
SKIP_STORAGE=false
|
||||||
|
|
||||||
# Derived (computed after argument parsing)
|
# Derived (computed after argument parsing)
|
||||||
SUBNET_BASE="" # e.g. 10.10.0
|
SUBNET_BASE="" # e.g. 10.10.0
|
||||||
|
|
@ -71,7 +72,12 @@ DETECT_VMBR1_IP="" # e.g. "10.10.0.1/24" or empty
|
||||||
DETECT_IP_FORWARD="" # "1" or "0"
|
DETECT_IP_FORWARD="" # "1" or "0"
|
||||||
DETECT_NAT_ACTIVE="" # "yes" or empty
|
DETECT_NAT_ACTIVE="" # "yes" or empty
|
||||||
|
|
||||||
|
# Storage
|
||||||
|
STATE_STORAGE="done" # done | available | needed
|
||||||
DETECT_ZPOOLS=""
|
DETECT_ZPOOLS=""
|
||||||
|
DETECT_PVE_STORAGE=""
|
||||||
|
DETECT_AVAILABLE_DISKS=""
|
||||||
|
DETECT_AVAILABLE_DISK_COUNT="0"
|
||||||
|
|
||||||
# WireGuard
|
# WireGuard
|
||||||
STATE_WIREGUARD="needed" # done | needed
|
STATE_WIREGUARD="needed" # done | needed
|
||||||
|
|
@ -85,16 +91,11 @@ STATE_FIREWALL="needed" # done | needed
|
||||||
DETECT_NFT_RULES="" # "yes" if nftables rules loaded with our pattern
|
DETECT_NFT_RULES="" # "yes" if nftables rules loaded with our pattern
|
||||||
DETECT_NFT_FILE="" # "yes" if config file exists
|
DETECT_NFT_FILE="" # "yes" if config file exists
|
||||||
|
|
||||||
# DNS (dnsmasq on vmbr1 -- required for IncusOS boot)
|
|
||||||
STATE_DNSMASQ="needed" # done | needed
|
|
||||||
DETECT_DNSMASQ_ACTIVE="" # "yes" if dnsmasq is installed and listening on vmbr1
|
|
||||||
|
|
||||||
# API token
|
# API token
|
||||||
STATE_API="needed" # done | partial | needed
|
STATE_API="needed" # done | partial | needed
|
||||||
DETECT_API_ROLE="" # "yes" or empty
|
DETECT_API_ROLE="" # "yes" or empty
|
||||||
DETECT_API_POOL="" # "yes" or empty
|
DETECT_API_POOL="" # "yes" or empty
|
||||||
DETECT_API_TOKEN="" # "yes" or empty
|
DETECT_API_TOKEN="" # "yes" or empty
|
||||||
DETECT_API_ACLS="" # "yes" if /vms ACL is set
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Color and formatting
|
# Color and formatting
|
||||||
|
|
@ -125,9 +126,9 @@ detail() { [[ "$VERBOSE" != true ]] && return; echo -e "${DIM} $*${RES
|
||||||
dry_run_guard() {
|
dry_run_guard() {
|
||||||
if [[ "$DRY_RUN" == true ]]; then
|
if [[ "$DRY_RUN" == true ]]; then
|
||||||
info "(dry-run) $*"
|
info "(dry-run) $*"
|
||||||
return 0
|
return 1
|
||||||
fi
|
fi
|
||||||
return 1
|
return 0
|
||||||
}
|
}
|
||||||
|
|
||||||
# Status indicator for the dashboard
|
# Status indicator for the dashboard
|
||||||
|
|
@ -205,6 +206,7 @@ ${BOLD}DESCRIPTION${RESET}
|
||||||
|
|
||||||
- No-subscription repositories and system update
|
- No-subscription repositories and system update
|
||||||
- Private bridge (vmbr1) with NAT masquerading
|
- Private bridge (vmbr1) with NAT masquerading
|
||||||
|
- ZFS storage pool on extra disks
|
||||||
- WireGuard tunnel for secure remote access
|
- WireGuard tunnel for secure remote access
|
||||||
- nftables firewall lockdown
|
- nftables firewall lockdown
|
||||||
- API token for incusos-proxmox automation
|
- API token for incusos-proxmox automation
|
||||||
|
|
@ -218,6 +220,7 @@ ${BOLD}OPTIONS${RESET}
|
||||||
${BOLD}--wg-subnet CIDR${RESET} WireGuard subnet (default: ${WG_SUBNET})
|
${BOLD}--wg-subnet CIDR${RESET} WireGuard subnet (default: ${WG_SUBNET})
|
||||||
${BOLD}--wg-port PORT${RESET} WireGuard listen port (default: ${WG_PORT})
|
${BOLD}--wg-port PORT${RESET} WireGuard listen port (default: ${WG_PORT})
|
||||||
${BOLD}--skip-repos${RESET} Skip repository changes
|
${BOLD}--skip-repos${RESET} Skip repository changes
|
||||||
|
${BOLD}--skip-storage${RESET} Skip storage pool creation
|
||||||
${BOLD}--skip-wireguard${RESET} Skip WireGuard setup
|
${BOLD}--skip-wireguard${RESET} Skip WireGuard setup
|
||||||
${BOLD}--skip-firewall${RESET} Skip firewall lockdown
|
${BOLD}--skip-firewall${RESET} Skip firewall lockdown
|
||||||
${BOLD}-n, --dry-run${RESET} Preview actions without executing
|
${BOLD}-n, --dry-run${RESET} Preview actions without executing
|
||||||
|
|
@ -247,7 +250,7 @@ ${BOLD}PREREQUISITES${RESET}
|
||||||
|
|
||||||
${BOLD}SEE ALSO${RESET}
|
${BOLD}SEE ALSO${RESET}
|
||||||
hetzner/hetzner-setup.md Manual installation guide
|
hetzner/hetzner-setup.md Manual installation guide
|
||||||
envs/hetzner/ Target config files
|
incusos/targets/hetzner/ Target config files
|
||||||
EOF
|
EOF
|
||||||
exit 0
|
exit 0
|
||||||
}
|
}
|
||||||
|
|
@ -279,6 +282,10 @@ parse_args() {
|
||||||
SKIP_REPOS=true
|
SKIP_REPOS=true
|
||||||
shift
|
shift
|
||||||
;;
|
;;
|
||||||
|
--skip-storage)
|
||||||
|
SKIP_STORAGE=true
|
||||||
|
shift
|
||||||
|
;;
|
||||||
--skip-wireguard)
|
--skip-wireguard)
|
||||||
SKIP_WIREGUARD=true
|
SKIP_WIREGUARD=true
|
||||||
shift
|
shift
|
||||||
|
|
@ -363,129 +370,65 @@ detect_all() {
|
||||||
fi
|
fi
|
||||||
success "SSH connection to ${SSH_HOST}"
|
success "SSH connection to ${SSH_HOST}"
|
||||||
|
|
||||||
# Gather everything in a single SSH call for speed.
|
# Gather everything in a single SSH call for speed
|
||||||
# Uses a quoted heredoc (<<'DETECT') so nothing is expanded locally --
|
|
||||||
# all $, ", etc. are passed literally to the remote shell.
|
|
||||||
local detection_blob
|
local detection_blob
|
||||||
detection_blob=$(ssh -o ConnectTimeout=10 -o BatchMode=yes "$SSH_HOST" bash <<'DETECT' 2>/dev/null || true
|
detection_blob=$(remote_output "
|
||||||
echo '---HOSTNAME---'
|
echo '---HOSTNAME---'
|
||||||
hostname
|
hostname
|
||||||
echo '---KERNEL---'
|
echo '---KERNEL---'
|
||||||
uname -r
|
uname -r
|
||||||
echo '---PVE_VERSION---'
|
echo '---PVE_VERSION---'
|
||||||
pvesh get /version --output-format json 2>/dev/null \
|
pvesh get /version --output-format json 2>/dev/null | python3 -c 'import sys,json; print(json.load(sys.stdin)[\"version\"])' 2>/dev/null || echo 'unknown'
|
||||||
| python3 -c 'import sys,json; print(json.load(sys.stdin)["version"])' 2>/dev/null \
|
echo '---PUBLIC_IP---'
|
||||||
|| echo 'unknown'
|
hostname -I | awk '{print \$1}'
|
||||||
echo '---PUBLIC_IP---'
|
echo '---BLOCK_DEVICES---'
|
||||||
hostname -I | awk '{print $1}'
|
lsblk -d -o NAME,SIZE,MODEL,ROTA,TYPE 2>/dev/null
|
||||||
echo '---BLOCK_DEVICES---'
|
echo '---ZPOOLS---'
|
||||||
lsblk -d -o NAME,SIZE,MODEL,ROTA,TYPE 2>/dev/null
|
zpool list -H 2>/dev/null || true
|
||||||
echo '---ZPOOLS---'
|
echo '---PVE_STORAGE---'
|
||||||
zpool list -H 2>/dev/null || true
|
pvesm status 2>/dev/null || true
|
||||||
echo '---ENTERPRISE_REPOS---'
|
echo '---ENTERPRISE_REPOS---'
|
||||||
# Check both .list (PVE 8) and .sources/deb822 (PVE 9) formats
|
grep -rl '^deb.*enterprise' /etc/apt/sources.list.d/*.list 2>/dev/null || echo 'none'
|
||||||
enterprise_active='no'
|
echo '---NOSUB_REPO---'
|
||||||
# Old .list format: uncommented deb lines with enterprise
|
test -f /etc/apt/sources.list.d/pve-no-subscription.list && echo 'yes' || echo 'no'
|
||||||
if grep -rl '^deb.*enterprise' /etc/apt/sources.list.d/*.list 2>/dev/null | head -1 | grep -q .; then
|
echo '---NAG_PATCHED---'
|
||||||
enterprise_active='yes'
|
grep -q 'void({' /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js 2>/dev/null && echo 'yes' || echo 'no'
|
||||||
fi
|
echo '---UPGRADABLE---'
|
||||||
# New .sources/deb822 format: check for enterprise files that are NOT disabled
|
apt list --upgradable 2>/dev/null | grep -c upgradable || echo '0'
|
||||||
for f in /etc/apt/sources.list.d/*enterprise*.sources /etc/apt/sources.list.d/*enterprise*.list; do
|
echo '---VMBR1---'
|
||||||
[ -f "$f" ] || continue
|
ip -br addr show vmbr1 2>/dev/null || echo 'none'
|
||||||
# deb822: Enabled: false means disabled
|
echo '---IP_FORWARD---'
|
||||||
if grep -qi '^Enabled:.*false' "$f" 2>/dev/null; then
|
cat /proc/sys/net/ipv4/ip_forward 2>/dev/null || echo '0'
|
||||||
continue
|
echo '---NAT_ACTIVE---'
|
||||||
fi
|
iptables -t nat -L POSTROUTING -n 2>/dev/null | grep -q MASQUERADE && echo 'yes' || echo 'no'
|
||||||
# deb822: if it has enterprise in URIs and no Enabled: false, it's active
|
echo '---WG_INSTALLED---'
|
||||||
if grep -qi 'enterprise' "$f" 2>/dev/null; then
|
command -v wg >/dev/null 2>&1 && echo 'yes' || echo 'no'
|
||||||
enterprise_active='yes'
|
echo '---WG_ACTIVE---'
|
||||||
fi
|
wg show wg0 2>/dev/null | head -1 || echo 'none'
|
||||||
done
|
echo '---WG_PEERS---'
|
||||||
echo "$enterprise_active"
|
wg show wg0 peers 2>/dev/null | wc -l || echo '0'
|
||||||
echo '---NOSUB_REPO---'
|
echo '---WG_HANDSHAKE---'
|
||||||
# Check both .list and .sources formats for no-subscription repo
|
wg show wg0 latest-handshakes 2>/dev/null | head -1 || echo 'none'
|
||||||
nosub='no'
|
echo '---NFT_FILE---'
|
||||||
if grep -rq 'pve-no-subscription' /etc/apt/sources.list.d/ 2>/dev/null; then
|
test -f /etc/nftables-hetzner.conf && echo 'yes' || echo 'no'
|
||||||
nosub='yes'
|
echo '---NFT_RULES---'
|
||||||
fi
|
nft list ruleset 2>/dev/null | grep -q 'nft-drop' && echo 'yes' || echo 'no'
|
||||||
echo "$nosub"
|
echo '---API_ROLE---'
|
||||||
echo '---NAG_PATCHED---'
|
pveum role list --output-format json 2>/dev/null | python3 -c 'import sys,json; any(r[\"roleid\"]==\"IncusOSDeployer\" for r in json.load(sys.stdin)) and print(\"yes\")' 2>/dev/null || echo 'no'
|
||||||
# Detect whether the subscription nag popup has been removed.
|
echo '---API_POOL---'
|
||||||
# Methods vary by tool and PVE version:
|
pveum pool list --output-format json 2>/dev/null | python3 -c 'import sys,json; any(p[\"poolid\"]==\"IncusLab\" for p in json.load(sys.stdin)) and print(\"yes\")' 2>/dev/null || echo 'no'
|
||||||
# 1. community-scripts post-install: installs a dpkg hook that patches
|
echo '---API_TOKEN---'
|
||||||
# proxmoxlib.js after every update, changing "!== 'active'" to
|
pveum user token list automation@pve --output-format json 2>/dev/null | python3 -c 'import sys,json; any(t[\"tokenid\"]==\"deploy\" for t in json.load(sys.stdin)) and print(\"yes\")' 2>/dev/null || echo 'no'
|
||||||
# "== 'NoMoreNagging'" on data.status lines. Leaves behind:
|
echo '---AVAILABLE_DISKS---'
|
||||||
# - /etc/apt/apt.conf.d/no-nag-script (dpkg hook)
|
used_disks=\$(zpool status 2>/dev/null | grep -oP '/dev/\\S+' | sed 's|/dev/||' || true)
|
||||||
# - /usr/local/bin/pve-remove-nag.sh (the patcher)
|
lsblk -dno NAME,SIZE,TYPE | while read name size type; do
|
||||||
# 2. pve-nag-buster: similar dpkg hook approach
|
[ \"\$type\" != 'disk' ] && continue
|
||||||
# 3. Manual sed: replaces Ext.Msg.show with void({
|
echo \"\$used_disks\" | grep -qw \"\$name\" && continue
|
||||||
nag_patched='no'
|
lsblk -no MOUNTPOINT /dev/\$name 2>/dev/null | grep -q '/' && continue
|
||||||
# Check for persistent dpkg hook (community-scripts / pve-nag-buster)
|
echo \"\$name \$size\"
|
||||||
if [ -f /etc/apt/apt.conf.d/no-nag-script ] || [ -f /usr/local/bin/pve-remove-nag.sh ]; then
|
done
|
||||||
nag_patched='yes'
|
echo '---END---'
|
||||||
# Check for pve-nag-buster package hook
|
" || true)
|
||||||
elif ls /etc/dpkg/dpkg.cfg.d/*nag* /etc/apt/apt.conf.d/*nag-buster* 2>/dev/null | grep -q .; then
|
|
||||||
nag_patched='yes'
|
|
||||||
# Check JS for void({ patch (manual sed method)
|
|
||||||
elif grep -q 'void({' /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js 2>/dev/null; then
|
|
||||||
nag_patched='yes'
|
|
||||||
# Check if nag text was removed entirely
|
|
||||||
elif ! grep -q "No valid subscription" /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js 2>/dev/null; then
|
|
||||||
nag_patched='yes'
|
|
||||||
fi
|
|
||||||
echo "$nag_patched"
|
|
||||||
echo '---UPGRADABLE---'
|
|
||||||
apt list --upgradable 2>/dev/null | grep -c upgradable || true
|
|
||||||
echo '---VMBR1---'
|
|
||||||
ip -br addr show vmbr1 2>/dev/null || echo 'none'
|
|
||||||
echo '---IP_FORWARD---'
|
|
||||||
cat /proc/sys/net/ipv4/ip_forward 2>/dev/null || echo '0'
|
|
||||||
echo '---NAT_ACTIVE---'
|
|
||||||
iptables -t nat -L POSTROUTING -n 2>/dev/null | grep -q MASQUERADE && echo 'yes' || echo 'no'
|
|
||||||
echo '---WG_INSTALLED---'
|
|
||||||
command -v wg >/dev/null 2>&1 && echo 'yes' || echo 'no'
|
|
||||||
echo '---WG_ACTIVE---'
|
|
||||||
wg_out=$(wg show wg0 2>/dev/null | head -1) || true
|
|
||||||
if [ -n "$wg_out" ]; then echo "$wg_out"; else echo 'none'; fi
|
|
||||||
echo '---WG_PEERS---'
|
|
||||||
wg_peers=$(wg show wg0 peers 2>/dev/null | wc -l) || true
|
|
||||||
echo "${wg_peers:-0}"
|
|
||||||
echo '---WG_HANDSHAKE---'
|
|
||||||
wg_hs=$(wg show wg0 latest-handshakes 2>/dev/null | head -1) || true
|
|
||||||
if [ -n "$wg_hs" ]; then echo "$wg_hs"; else echo 'none'; fi
|
|
||||||
echo '---NFT_FILE---'
|
|
||||||
test -f /etc/nftables-hetzner.conf && echo 'yes' || echo 'no'
|
|
||||||
echo '---NFT_RULES---'
|
|
||||||
nft list ruleset 2>/dev/null | grep -q 'nft-drop' && echo 'yes' || echo 'no'
|
|
||||||
echo '---DNSMASQ---'
|
|
||||||
# dnsmasq installed and listening on port 53 on vmbr1?
|
|
||||||
dnsmasq_ok='no'
|
|
||||||
if command -v dnsmasq >/dev/null 2>&1 && systemctl is-active dnsmasq >/dev/null 2>&1; then
|
|
||||||
# Check it's listening on the bridge IP
|
|
||||||
if ss -ulnp 2>/dev/null | grep ':53 ' | grep -q dnsmasq; then
|
|
||||||
dnsmasq_ok='yes'
|
|
||||||
fi
|
|
||||||
fi
|
|
||||||
echo "$dnsmasq_ok"
|
|
||||||
echo '---API_ROLE---'
|
|
||||||
pveum role list --output-format json 2>/dev/null \
|
|
||||||
| python3 -c 'import sys,json; print("yes" if any(r["roleid"]=="IncusOSDeployer" for r in json.load(sys.stdin)) else "no")' 2>/dev/null \
|
|
||||||
|| echo 'no'
|
|
||||||
echo '---API_POOL---'
|
|
||||||
pveum pool list --output-format json 2>/dev/null \
|
|
||||||
| python3 -c 'import sys,json; print("yes" if any(p["poolid"]=="IncusLab" for p in json.load(sys.stdin)) else "no")' 2>/dev/null \
|
|
||||||
|| echo 'no'
|
|
||||||
echo '---API_TOKEN---'
|
|
||||||
pveum user token list automation@pve --output-format json 2>/dev/null \
|
|
||||||
| python3 -c 'import sys,json; print("yes" if any(t["tokenid"]=="deploy" for t in json.load(sys.stdin)) else "no")' 2>/dev/null \
|
|
||||||
|| echo 'no'
|
|
||||||
echo '---API_ACLS---'
|
|
||||||
pveum acl list --output-format json 2>/dev/null \
|
|
||||||
| python3 -c 'import sys,json; acls=json.load(sys.stdin); paths={a.get("path") for a in acls if "automation@pve" in a.get("ugid","")}; ok=("/vms" in paths and "/sdn" in paths); print("yes" if ok else "no")' 2>/dev/null \
|
|
||||||
|| echo 'no'
|
|
||||||
echo '---END---'
|
|
||||||
DETECT
|
|
||||||
)
|
|
||||||
|
|
||||||
# Parse the blob into variables
|
# Parse the blob into variables
|
||||||
local section=""
|
local section=""
|
||||||
|
|
@ -508,7 +451,7 @@ ${line}"
|
||||||
fi
|
fi
|
||||||
done <<< "$detection_blob"
|
done <<< "$detection_blob"
|
||||||
# Process last section
|
# Process last section
|
||||||
if [[ -n "$section" ]]; then _parse_section "$section" "$section_lines"; fi
|
[[ -n "$section" ]] && _parse_section "$section" "$section_lines"
|
||||||
|
|
||||||
# Determine composite states
|
# Determine composite states
|
||||||
_compute_states
|
_compute_states
|
||||||
|
|
@ -526,11 +469,12 @@ _parse_section() {
|
||||||
PUBLIC_IP) DETECT_PUBLIC_IP="$content" ;;
|
PUBLIC_IP) DETECT_PUBLIC_IP="$content" ;;
|
||||||
BLOCK_DEVICES) DETECT_BLOCK_DEVICES="$content" ;;
|
BLOCK_DEVICES) DETECT_BLOCK_DEVICES="$content" ;;
|
||||||
ZPOOLS) DETECT_ZPOOLS="$content" ;;
|
ZPOOLS) DETECT_ZPOOLS="$content" ;;
|
||||||
|
PVE_STORAGE) DETECT_PVE_STORAGE="$content" ;;
|
||||||
ENTERPRISE_REPOS)
|
ENTERPRISE_REPOS)
|
||||||
if [[ "$content" == "yes" ]]; then
|
if [[ "$content" == "none" ]]; then
|
||||||
DETECT_ENTERPRISE_ACTIVE="yes"
|
|
||||||
else
|
|
||||||
DETECT_ENTERPRISE_ACTIVE=""
|
DETECT_ENTERPRISE_ACTIVE=""
|
||||||
|
else
|
||||||
|
DETECT_ENTERPRISE_ACTIVE="$content"
|
||||||
fi
|
fi
|
||||||
;;
|
;;
|
||||||
NOSUB_REPO) DETECT_NOSUB_EXISTS="$content" ;;
|
NOSUB_REPO) DETECT_NOSUB_EXISTS="$content" ;;
|
||||||
|
|
@ -545,32 +489,34 @@ _parse_section() {
|
||||||
NAT_ACTIVE) DETECT_NAT_ACTIVE="$content" ;;
|
NAT_ACTIVE) DETECT_NAT_ACTIVE="$content" ;;
|
||||||
WG_INSTALLED) DETECT_WG_INSTALLED="$content" ;;
|
WG_INSTALLED) DETECT_WG_INSTALLED="$content" ;;
|
||||||
WG_ACTIVE)
|
WG_ACTIVE)
|
||||||
if [[ "$content" != "none" ]] && [[ -n "$content" ]]; then
|
if [[ "$content" != "none" ]]; then
|
||||||
DETECT_WG_ACTIVE="yes"
|
DETECT_WG_ACTIVE="yes"
|
||||||
fi
|
fi
|
||||||
;;
|
;;
|
||||||
WG_PEERS) DETECT_WG_PEERS="${content//[^0-9]/}" ;;
|
WG_PEERS) DETECT_WG_PEERS="${content//[^0-9]/}" ;;
|
||||||
WG_HANDSHAKE)
|
WG_HANDSHAKE)
|
||||||
if [[ "$content" != "none" ]] && [[ -n "$content" ]]; then
|
if [[ "$content" != "none" ]]; then
|
||||||
DETECT_WG_HANDSHAKE="$content"
|
DETECT_WG_HANDSHAKE="$content"
|
||||||
fi
|
fi
|
||||||
;;
|
;;
|
||||||
NFT_FILE) DETECT_NFT_FILE="$content" ;;
|
NFT_FILE) DETECT_NFT_FILE="$content" ;;
|
||||||
NFT_RULES) DETECT_NFT_RULES="$content" ;;
|
NFT_RULES) DETECT_NFT_RULES="$content" ;;
|
||||||
DNSMASQ)
|
|
||||||
if [[ "$content" == "yes" ]]; then DETECT_DNSMASQ_ACTIVE="yes"; fi
|
|
||||||
;;
|
|
||||||
API_ROLE)
|
API_ROLE)
|
||||||
if [[ "$content" == "yes" ]]; then DETECT_API_ROLE="yes"; fi
|
[[ "$content" == "yes" ]] && DETECT_API_ROLE="yes"
|
||||||
;;
|
;;
|
||||||
API_POOL)
|
API_POOL)
|
||||||
if [[ "$content" == "yes" ]]; then DETECT_API_POOL="yes"; fi
|
[[ "$content" == "yes" ]] && DETECT_API_POOL="yes"
|
||||||
;;
|
;;
|
||||||
API_TOKEN)
|
API_TOKEN)
|
||||||
if [[ "$content" == "yes" ]]; then DETECT_API_TOKEN="yes"; fi
|
[[ "$content" == "yes" ]] && DETECT_API_TOKEN="yes"
|
||||||
;;
|
;;
|
||||||
API_ACLS)
|
AVAILABLE_DISKS)
|
||||||
if [[ "$content" == "yes" ]]; then DETECT_API_ACLS="yes"; fi
|
DETECT_AVAILABLE_DISKS="$content"
|
||||||
|
if [[ -n "$content" ]]; then
|
||||||
|
DETECT_AVAILABLE_DISK_COUNT=$(echo "$content" | wc -l)
|
||||||
|
else
|
||||||
|
DETECT_AVAILABLE_DISK_COUNT="0"
|
||||||
|
fi
|
||||||
;;
|
;;
|
||||||
esac
|
esac
|
||||||
}
|
}
|
||||||
|
|
@ -603,6 +549,21 @@ _compute_states() {
|
||||||
STATE_BRIDGE="needed"
|
STATE_BRIDGE="needed"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
# Storage: check if any ZFS pool is registered as PVE storage
|
||||||
|
local zfs_in_pve
|
||||||
|
zfs_in_pve=$(echo "$DETECT_PVE_STORAGE" | grep -c "zfspool" || true)
|
||||||
|
if [[ "$zfs_in_pve" -gt 0 ]]; then
|
||||||
|
if [[ "$DETECT_AVAILABLE_DISK_COUNT" -gt 0 ]]; then
|
||||||
|
STATE_STORAGE="available" # has pool but also unused disks
|
||||||
|
else
|
||||||
|
STATE_STORAGE="done"
|
||||||
|
fi
|
||||||
|
elif [[ "$DETECT_AVAILABLE_DISK_COUNT" -gt 0 ]]; then
|
||||||
|
STATE_STORAGE="needed"
|
||||||
|
else
|
||||||
|
STATE_STORAGE="done" # no pool, but no disks available either
|
||||||
|
fi
|
||||||
|
|
||||||
# WireGuard
|
# WireGuard
|
||||||
if [[ "$DETECT_WG_ACTIVE" == "yes" ]]; then
|
if [[ "$DETECT_WG_ACTIVE" == "yes" ]]; then
|
||||||
STATE_WIREGUARD="done"
|
STATE_WIREGUARD="done"
|
||||||
|
|
@ -618,20 +579,13 @@ _compute_states() {
|
||||||
fi
|
fi
|
||||||
|
|
||||||
# API token
|
# API token
|
||||||
if [[ "$DETECT_API_ROLE" == "yes" ]] && [[ "$DETECT_API_POOL" == "yes" ]] && [[ "$DETECT_API_TOKEN" == "yes" ]] && [[ "$DETECT_API_ACLS" == "yes" ]]; then
|
if [[ "$DETECT_API_ROLE" == "yes" ]] && [[ "$DETECT_API_POOL" == "yes" ]] && [[ "$DETECT_API_TOKEN" == "yes" ]]; then
|
||||||
STATE_API="done"
|
STATE_API="done"
|
||||||
elif [[ "$DETECT_API_ROLE" == "yes" ]] || [[ "$DETECT_API_POOL" == "yes" ]] || [[ "$DETECT_API_TOKEN" == "yes" ]]; then
|
elif [[ "$DETECT_API_ROLE" == "yes" ]] || [[ "$DETECT_API_POOL" == "yes" ]] || [[ "$DETECT_API_TOKEN" == "yes" ]]; then
|
||||||
STATE_API="partial"
|
STATE_API="partial"
|
||||||
else
|
else
|
||||||
STATE_API="needed"
|
STATE_API="needed"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
# dnsmasq (DNS for VMs on vmbr1 -- required for IncusOS boot)
|
|
||||||
if [[ "$DETECT_DNSMASQ_ACTIVE" == "yes" ]]; then
|
|
||||||
STATE_DNSMASQ="done"
|
|
||||||
else
|
|
||||||
STATE_DNSMASQ="needed"
|
|
||||||
fi
|
|
||||||
}
|
}
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
@ -680,12 +634,14 @@ show_status() {
|
||||||
|
|
||||||
local effective_repos="$STATE_REPOS"
|
local effective_repos="$STATE_REPOS"
|
||||||
local effective_update="$STATE_UPDATE"
|
local effective_update="$STATE_UPDATE"
|
||||||
|
local effective_storage="$STATE_STORAGE"
|
||||||
local effective_wg="$STATE_WIREGUARD"
|
local effective_wg="$STATE_WIREGUARD"
|
||||||
local effective_fw="$STATE_FIREWALL"
|
local effective_fw="$STATE_FIREWALL"
|
||||||
|
|
||||||
if [[ "$SKIP_REPOS" == true ]]; then effective_repos="skipped"; effective_update="skipped"; fi
|
[[ "$SKIP_REPOS" == true ]] && effective_repos="skipped" && effective_update="skipped"
|
||||||
if [[ "$SKIP_WIREGUARD" == true ]]; then effective_wg="skipped"; fi
|
[[ "$SKIP_STORAGE" == true ]] && effective_storage="skipped"
|
||||||
if [[ "$SKIP_FIREWALL" == true ]]; then effective_fw="skipped"; fi
|
[[ "$SKIP_WIREGUARD" == true ]] && effective_wg="skipped"
|
||||||
|
[[ "$SKIP_FIREWALL" == true ]] && effective_fw="skipped"
|
||||||
|
|
||||||
printf " %-20s %b" "Repositories" "$(status_icon "$effective_repos")"
|
printf " %-20s %b" "Repositories" "$(status_icon "$effective_repos")"
|
||||||
case "$effective_repos" in
|
case "$effective_repos" in
|
||||||
|
|
@ -716,6 +672,14 @@ show_status() {
|
||||||
needed) echo " vmbr1 not yet created" ;;
|
needed) echo " vmbr1 not yet created" ;;
|
||||||
esac
|
esac
|
||||||
|
|
||||||
|
printf " %-20s %b" "Storage" "$(status_icon "$effective_storage")"
|
||||||
|
case "$effective_storage" in
|
||||||
|
done) echo " ZFS pool registered" ;;
|
||||||
|
available) echo " ZFS pool exists, ${DETECT_AVAILABLE_DISK_COUNT} unused disk(s) available" ;;
|
||||||
|
needed) echo " ${DETECT_AVAILABLE_DISK_COUNT} unused disk(s), no pool created" ;;
|
||||||
|
skipped) echo "" ;;
|
||||||
|
esac
|
||||||
|
|
||||||
printf " %-20s %b" "WireGuard" "$(status_icon "$effective_wg")"
|
printf " %-20s %b" "WireGuard" "$(status_icon "$effective_wg")"
|
||||||
case "$effective_wg" in
|
case "$effective_wg" in
|
||||||
done) echo " wg0 active, ${DETECT_WG_PEERS:-0} peer(s)" ;;
|
done) echo " wg0 active, ${DETECT_WG_PEERS:-0} peer(s)" ;;
|
||||||
|
|
@ -734,9 +698,9 @@ show_status() {
|
||||||
done) echo " nftables rules active" ;;
|
done) echo " nftables rules active" ;;
|
||||||
needed)
|
needed)
|
||||||
if [[ "$DETECT_NFT_FILE" == "yes" ]]; then
|
if [[ "$DETECT_NFT_FILE" == "yes" ]]; then
|
||||||
echo " config written, apply manually (see firewall step)"
|
echo " config file exists but rules not loaded"
|
||||||
else
|
else
|
||||||
echo " config not yet written"
|
echo " no rules configured"
|
||||||
fi
|
fi
|
||||||
;;
|
;;
|
||||||
skipped) echo "" ;;
|
skipped) echo "" ;;
|
||||||
|
|
@ -755,23 +719,17 @@ show_status() {
|
||||||
needed) echo " not configured" ;;
|
needed) echo " not configured" ;;
|
||||||
esac
|
esac
|
||||||
|
|
||||||
printf " %-20s %b" "DNS (dnsmasq)" "$(status_icon "$STATE_DNSMASQ")"
|
|
||||||
case "$STATE_DNSMASQ" in
|
|
||||||
done) echo " dnsmasq active, forwarding DNS on vmbr1" ;;
|
|
||||||
needed) echo " not installed (required for IncusOS boot)" ;;
|
|
||||||
esac
|
|
||||||
|
|
||||||
echo ""
|
echo ""
|
||||||
|
|
||||||
# Count pending items
|
# Count pending items
|
||||||
local pending=0
|
local pending=0
|
||||||
if [[ "$effective_repos" == "needed" || "$effective_repos" == "partial" ]]; then pending=$((pending + 1)); fi
|
[[ "$effective_repos" == "needed" || "$effective_repos" == "partial" ]] && pending=$((pending + 1))
|
||||||
if [[ "$effective_update" == "needed" ]]; then pending=$((pending + 1)); fi
|
[[ "$effective_update" == "needed" ]] && pending=$((pending + 1))
|
||||||
if [[ "$STATE_BRIDGE" == "needed" || "$STATE_BRIDGE" == "partial" ]]; then pending=$((pending + 1)); fi
|
[[ "$STATE_BRIDGE" == "needed" || "$STATE_BRIDGE" == "partial" ]] && pending=$((pending + 1))
|
||||||
if [[ "$effective_wg" == "needed" ]]; then pending=$((pending + 1)); fi
|
[[ "$effective_storage" == "needed" ]] && pending=$((pending + 1))
|
||||||
if [[ "$effective_fw" == "needed" ]]; then pending=$((pending + 1)); fi
|
[[ "$effective_wg" == "needed" ]] && pending=$((pending + 1))
|
||||||
if [[ "$STATE_API" == "needed" || "$STATE_API" == "partial" ]]; then pending=$((pending + 1)); fi
|
[[ "$effective_fw" == "needed" ]] && pending=$((pending + 1))
|
||||||
if [[ "$STATE_DNSMASQ" == "needed" ]]; then pending=$((pending + 1)); fi
|
[[ "$STATE_API" == "needed" || "$STATE_API" == "partial" ]] && pending=$((pending + 1))
|
||||||
|
|
||||||
if [[ "$pending" -eq 0 ]]; then
|
if [[ "$pending" -eq 0 ]]; then
|
||||||
success "All configured -- nothing to do"
|
success "All configured -- nothing to do"
|
||||||
|
|
@ -818,30 +776,13 @@ apply_repos() {
|
||||||
fi
|
fi
|
||||||
|
|
||||||
if [[ -n "$DETECT_ENTERPRISE_ACTIVE" ]]; then
|
if [[ -n "$DETECT_ENTERPRISE_ACTIVE" ]]; then
|
||||||
# Handle both .list (PVE 8 / bookworm) and .sources (PVE 9 / trixie) formats
|
remote "sed -i 's/^deb/# deb/' /etc/apt/sources.list.d/pve-enterprise.list 2>/dev/null || true"
|
||||||
# .list format: comment out deb lines
|
remote "test -f /etc/apt/sources.list.d/ceph.list && sed -i 's/^deb/# deb/' /etc/apt/sources.list.d/ceph.list || true"
|
||||||
remote "for f in /etc/apt/sources.list.d/*enterprise*.list; do [ -f \"\$f\" ] && sed -i 's/^deb/# deb/' \"\$f\"; done 2>/dev/null || true"
|
|
||||||
# .sources/deb822 format: set Enabled: false
|
|
||||||
remote "for f in /etc/apt/sources.list.d/*enterprise*.sources; do [ -f \"\$f\" ] && sed -i '/^Enabled:/d' \"\$f\" && echo 'Enabled: false' >> \"\$f\"; done 2>/dev/null || true"
|
|
||||||
success "Enterprise repos disabled"
|
success "Enterprise repos disabled"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
if [[ "$DETECT_NOSUB_EXISTS" != "yes" ]]; then
|
if [[ "$DETECT_NOSUB_EXISTS" != "yes" ]]; then
|
||||||
# Detect PVE version to choose format
|
remote "echo 'deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription' > /etc/apt/sources.list.d/pve-no-subscription.list"
|
||||||
local pve_major="${DETECT_PVE_VERSION%%.*}"
|
|
||||||
if [[ "${pve_major:-8}" -ge 9 ]]; then
|
|
||||||
# PVE 9+ uses deb822 .sources format
|
|
||||||
remote "cat > /etc/apt/sources.list.d/proxmox.sources <<'REPO'
|
|
||||||
Types: deb
|
|
||||||
URIs: http://download.proxmox.com/debian/pve
|
|
||||||
Suites: trixie
|
|
||||||
Components: pve-no-subscription
|
|
||||||
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
|
|
||||||
REPO"
|
|
||||||
else
|
|
||||||
# PVE 8 uses traditional .list format
|
|
||||||
remote "echo 'deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription' > /etc/apt/sources.list.d/pve-no-subscription.list"
|
|
||||||
fi
|
|
||||||
success "No-subscription repo added"
|
success "No-subscription repo added"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
|
@ -967,46 +908,94 @@ iface vmbr1 inet static
|
||||||
}
|
}
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Apply: dnsmasq (DNS for VMs on vmbr1)
|
# Apply: storage
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
apply_dnsmasq() {
|
apply_storage() {
|
||||||
if [[ "$STATE_DNSMASQ" == "done" ]]; then
|
if [[ "$SKIP_STORAGE" == true ]] || [[ "$STATE_STORAGE" == "done" ]]; then
|
||||||
return 0
|
return 0
|
||||||
fi
|
fi
|
||||||
|
|
||||||
step "DNS (dnsmasq on vmbr1)"
|
step "ZFS storage pool"
|
||||||
|
|
||||||
info "Will: install dnsmasq, configure it to forward DNS queries from VMs on vmbr1"
|
if [[ "$STATE_STORAGE" == "available" ]]; then
|
||||||
info " Interface: vmbr1 (${SUBNET_GW})"
|
info "ZFS pool already registered with Proxmox"
|
||||||
info " Upstream: 1.1.1.1, 8.8.8.8"
|
info "${DETECT_AVAILABLE_DISK_COUNT} additional unused disk(s) available:"
|
||||||
info " Required for IncusOS boot (downloads updates from images.linuxcontainers.org)"
|
echo "$DETECT_AVAILABLE_DISKS" | while IFS= read -r line; do
|
||||||
|
[[ -n "$line" ]] && echo " /dev/$line"
|
||||||
|
done
|
||||||
|
if ! confirm "Create an additional storage pool?"; then
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
if ! confirm "Install and configure dnsmasq?"; then
|
if [[ "$STATE_STORAGE" == "needed" ]]; then
|
||||||
|
# Check if a zpool named local-zfs already exists but isn't in PVE
|
||||||
|
local existing_pool
|
||||||
|
existing_pool=$(remote_output "zpool list -H local-zfs 2>/dev/null" || true)
|
||||||
|
if [[ -n "$existing_pool" ]]; then
|
||||||
|
info "ZFS pool 'local-zfs' exists but not registered with Proxmox"
|
||||||
|
if confirm "Register local-zfs with Proxmox?"; then
|
||||||
|
if ! dry_run_guard "pvesm add zfspool local-zfs"; then
|
||||||
|
remote "pvesm add zfspool local-zfs -pool local-zfs"
|
||||||
|
remote "pvesm set local-zfs -content images,rootdir"
|
||||||
|
success "local-zfs registered"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ "$DETECT_AVAILABLE_DISK_COUNT" -eq 0 ]]; then
|
||||||
|
info "No unused disks found"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
info "Available disks:"
|
||||||
|
echo "$DETECT_AVAILABLE_DISKS" | while IFS= read -r line; do
|
||||||
|
[[ -n "$line" ]] && echo " /dev/$line"
|
||||||
|
done
|
||||||
|
|
||||||
|
local disk_count="$DETECT_AVAILABLE_DISK_COUNT"
|
||||||
|
local disk_names
|
||||||
|
disk_names=$(echo "$DETECT_AVAILABLE_DISKS" | awk 'NF{print "/dev/"$1}' | tr '\n' ' ')
|
||||||
|
|
||||||
|
local pool_type
|
||||||
|
if [[ "$disk_count" -eq 1 ]]; then
|
||||||
|
pool_type="single"
|
||||||
|
info "Recommendation: single disk (no redundancy)"
|
||||||
|
elif [[ "$disk_count" -eq 2 ]]; then
|
||||||
|
pool_type="mirror"
|
||||||
|
info "Recommendation: mirror (2 disks, full redundancy)"
|
||||||
|
else
|
||||||
|
pool_type="raidz1"
|
||||||
|
info "Recommendation: raidz1 (${disk_count} disks, 1-disk redundancy)"
|
||||||
|
fi
|
||||||
|
|
||||||
|
local pool_name
|
||||||
|
pool_name=$(prompt_value "Pool name" "local-zfs")
|
||||||
|
|
||||||
|
if ! confirm "Create ZFS pool '${pool_name}' (${pool_type}) with: ${disk_names}?"; then
|
||||||
info "Skipped"
|
info "Skipped"
|
||||||
return 0
|
return 0
|
||||||
fi
|
fi
|
||||||
|
|
||||||
dry_run_guard
|
if dry_run_guard "zpool create ${pool_name} ${pool_type} ${disk_names}"; then
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
|
||||||
remote "apt-get install -y dnsmasq"
|
local zpool_cmd="zpool create ${pool_name}"
|
||||||
|
case "$pool_type" in
|
||||||
|
mirror) zpool_cmd="${zpool_cmd} mirror ${disk_names}" ;;
|
||||||
|
raidz1) zpool_cmd="${zpool_cmd} raidz1 ${disk_names}" ;;
|
||||||
|
single) zpool_cmd="${zpool_cmd} ${disk_names}" ;;
|
||||||
|
esac
|
||||||
|
|
||||||
local dnsmasq_conf
|
remote "$zpool_cmd"
|
||||||
dnsmasq_conf="# DNS for VMs on vmbr1 (${SUBNET})
|
remote "pvesm add zfspool ${pool_name} -pool ${pool_name}"
|
||||||
# Generated by proxmox-setup -- do not edit manually
|
remote "pvesm set ${pool_name} -content images,rootdir"
|
||||||
interface=vmbr1
|
|
||||||
bind-interfaces
|
|
||||||
no-resolv
|
|
||||||
server=1.1.1.1
|
|
||||||
server=8.8.8.8
|
|
||||||
domain-needed
|
|
||||||
bogus-priv"
|
|
||||||
|
|
||||||
remote_write "/etc/dnsmasq.d/vmbr1.conf" "$dnsmasq_conf"
|
success "ZFS pool '${pool_name}' created and registered"
|
||||||
remote "systemctl enable dnsmasq && systemctl restart dnsmasq"
|
|
||||||
remote "ss -ulnp 2>/dev/null | grep ':53 ' | grep -q dnsmasq && echo 'OK' || echo 'WARN: dnsmasq not listening on port 53'"
|
|
||||||
|
|
||||||
success "dnsmasq configured on vmbr1"
|
|
||||||
}
|
}
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
@ -1030,21 +1019,6 @@ apply_wireguard() {
|
||||||
return 0
|
return 0
|
||||||
fi
|
fi
|
||||||
|
|
||||||
if [[ "$DRY_RUN" == true ]]; then
|
|
||||||
info "(dry-run) Client config will look like:"
|
|
||||||
echo ""
|
|
||||||
echo " [Interface]"
|
|
||||||
echo " PrivateKey = <generated on setup>"
|
|
||||||
echo " Address = ${WG_CLIENT_IP}/${WG_SUBNET_MASK}"
|
|
||||||
echo ""
|
|
||||||
echo " [Peer]"
|
|
||||||
echo " PublicKey = <server public key>"
|
|
||||||
echo " Endpoint = ${DETECT_PUBLIC_IP:-<server-public-ip>}:${WG_PORT}"
|
|
||||||
echo " AllowedIPs = 10.10.0.0/16"
|
|
||||||
echo " PersistentKeepalive = 25"
|
|
||||||
echo ""
|
|
||||||
fi
|
|
||||||
|
|
||||||
if dry_run_guard "Install and configure WireGuard (${WG_SERVER_IP}:${WG_PORT})"; then
|
if dry_run_guard "Install and configure WireGuard (${WG_SERVER_IP}:${WG_PORT})"; then
|
||||||
return 0
|
return 0
|
||||||
fi
|
fi
|
||||||
|
|
@ -1107,22 +1081,6 @@ AllowedIPs = ${WG_CLIENT_IP}/32"
|
||||||
echo " PersistentKeepalive = 25"
|
echo " PersistentKeepalive = 25"
|
||||||
echo ""
|
echo ""
|
||||||
warn "Save the client config now -- private key is not stored anywhere else"
|
warn "Save the client config now -- private key is not stored anywhere else"
|
||||||
echo ""
|
|
||||||
info "Verify WireGuard before applying firewall rules"
|
|
||||||
if confirm "Connect your WireGuard client now and verify the tunnel?"; then
|
|
||||||
info "Checking server-side WireGuard status..."
|
|
||||||
local wg_status
|
|
||||||
wg_status=$(remote_output "wg show wg0 2>/dev/null" || true)
|
|
||||||
if echo "$wg_status" | grep -q "latest handshake"; then
|
|
||||||
success "WireGuard is working -- handshake confirmed"
|
|
||||||
elif echo "$wg_status" | grep -q "peer:"; then
|
|
||||||
warn "Peer configured but no handshake yet -- client connected?"
|
|
||||||
info "Try: ping ${WG_SERVER_IP}"
|
|
||||||
else
|
|
||||||
warn "No peers visible -- check wg0 is up on server"
|
|
||||||
remote_output "systemctl status wg-quick@wg0 --no-pager -l" || true
|
|
||||||
fi
|
|
||||||
fi
|
|
||||||
}
|
}
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
@ -1136,6 +1094,31 @@ apply_firewall() {
|
||||||
|
|
||||||
step "Firewall lockdown (nftables)"
|
step "Firewall lockdown (nftables)"
|
||||||
|
|
||||||
|
if [[ "$DETECT_NFT_FILE" == "yes" ]]; then
|
||||||
|
info "Config file exists but rules not loaded"
|
||||||
|
if confirm "Load existing firewall rules from /etc/nftables-hetzner.conf?"; then
|
||||||
|
if ! dry_run_guard "nft -f /etc/nftables-hetzner.conf"; then
|
||||||
|
remote "nft -f /etc/nftables-hetzner.conf"
|
||||||
|
remote "systemctl enable nftables"
|
||||||
|
success "Firewall rules loaded"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
info "Will restrict public interface to SSH (22) and WireGuard (${WG_PORT}) only"
|
||||||
|
info "Web UI (8006) and VM traffic only via WireGuard"
|
||||||
|
warn "Make sure WireGuard is working before applying firewall rules!"
|
||||||
|
|
||||||
|
if ! confirm "Apply firewall rules?"; then
|
||||||
|
info "Skipped"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
if dry_run_guard "Apply nftables rules (SSH + WG only on public interface)"; then
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
|
||||||
local nft_rules="#!/usr/sbin/nft -f
|
local nft_rules="#!/usr/sbin/nft -f
|
||||||
|
|
||||||
flush ruleset
|
flush ruleset
|
||||||
|
|
@ -1187,26 +1170,12 @@ table inet filter {
|
||||||
}
|
}
|
||||||
}"
|
}"
|
||||||
|
|
||||||
if [[ "$DETECT_NFT_FILE" != "yes" ]]; then
|
remote_write "/etc/nftables-hetzner.conf" "$nft_rules"
|
||||||
if dry_run_guard "Write /etc/nftables-hetzner.conf on ${SSH_HOST}"; then
|
remote "nft -f /etc/nftables-hetzner.conf"
|
||||||
info "Rules that would be written to /etc/nftables-hetzner.conf:"
|
remote "cp /etc/nftables-hetzner.conf /etc/nftables.conf"
|
||||||
echo "$nft_rules" | sed 's/^/ /'
|
remote "systemctl enable nftables"
|
||||||
echo ""
|
|
||||||
else
|
|
||||||
remote_write "/etc/nftables-hetzner.conf" "$nft_rules"
|
|
||||||
success "Config written to /etc/nftables-hetzner.conf"
|
|
||||||
fi
|
|
||||||
else
|
|
||||||
info "Config already at /etc/nftables-hetzner.conf (rules not yet loaded)"
|
|
||||||
fi
|
|
||||||
|
|
||||||
warn "Firewall rules are NOT applied automatically — misfire locks out SSH on Hetzner permanently"
|
success "Firewall applied (SSH + WireGuard only on public interface)"
|
||||||
info "Once WireGuard is verified, apply manually:"
|
|
||||||
echo ""
|
|
||||||
echo " ssh ${SSH_HOST} nft -f /etc/nftables-hetzner.conf"
|
|
||||||
echo " ssh ${SSH_HOST} systemctl enable nftables"
|
|
||||||
echo " ssh ${SSH_HOST} nft list ruleset # verify"
|
|
||||||
echo ""
|
|
||||||
}
|
}
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
@ -1225,7 +1194,6 @@ apply_api_token() {
|
||||||
[[ "$DETECT_API_ROLE" == "yes" ]] || actions="create IncusOSDeployer role"
|
[[ "$DETECT_API_ROLE" == "yes" ]] || actions="create IncusOSDeployer role"
|
||||||
[[ "$DETECT_API_POOL" == "yes" ]] || actions="${actions:+$actions, }create IncusLab pool"
|
[[ "$DETECT_API_POOL" == "yes" ]] || actions="${actions:+$actions, }create IncusLab pool"
|
||||||
[[ "$DETECT_API_TOKEN" == "yes" ]] || actions="${actions:+$actions, }create automation@pve!deploy token"
|
[[ "$DETECT_API_TOKEN" == "yes" ]] || actions="${actions:+$actions, }create automation@pve!deploy token"
|
||||||
[[ "$DETECT_API_ACLS" == "yes" ]] || actions="${actions:+$actions, }add missing ACLs (/vms, /sdn, storage)"
|
|
||||||
|
|
||||||
info "Will: ${actions}"
|
info "Will: ${actions}"
|
||||||
|
|
||||||
|
|
@ -1242,8 +1210,9 @@ apply_api_token() {
|
||||||
if [[ "$DETECT_API_ROLE" != "yes" ]]; then
|
if [[ "$DETECT_API_ROLE" != "yes" ]]; then
|
||||||
remote "pveum role add IncusOSDeployer -privs \
|
remote "pveum role add IncusOSDeployer -privs \
|
||||||
'VM.Allocate VM.Config.CDROM VM.Config.CPU VM.Config.Disk VM.Config.HWType \
|
'VM.Allocate VM.Config.CDROM VM.Config.CPU VM.Config.Disk VM.Config.HWType \
|
||||||
VM.Config.Memory VM.Config.Network VM.Config.Options VM.PowerMgmt VM.Audit \
|
VM.Config.Memory VM.Config.Network VM.Config.Options VM.PowerMgmt \
|
||||||
Datastore.AllocateSpace Datastore.AllocateTemplate Datastore.Audit \
|
VM.Monitor VM.Audit VM.Console \
|
||||||
|
Datastore.AllocateSpace Datastore.Audit \
|
||||||
Pool.Allocate Pool.Audit \
|
Pool.Allocate Pool.Audit \
|
||||||
SDN.Use Sys.Modify'"
|
SDN.Use Sys.Modify'"
|
||||||
success "Created IncusOSDeployer role"
|
success "Created IncusOSDeployer role"
|
||||||
|
|
@ -1266,22 +1235,8 @@ apply_api_token() {
|
||||||
local token_secret
|
local token_secret
|
||||||
token_secret=$(echo "$token_output" | python3 -c "import sys,json; print(json.load(sys.stdin)['value'])" 2>/dev/null || echo "$token_output")
|
token_secret=$(echo "$token_output" | python3 -c "import sys,json; print(json.load(sys.stdin)['value'])" 2>/dev/null || echo "$token_output")
|
||||||
|
|
||||||
success "API token created"
|
# Assign ACLs
|
||||||
echo ""
|
|
||||||
warn "Token secret (save now -- shown only once):"
|
|
||||||
echo ""
|
|
||||||
echo " ${token_secret}"
|
|
||||||
echo ""
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Assign ACLs (idempotent -- run whenever any ACL is missing).
|
|
||||||
# PVE 9.x requires /sdn for bridge-backed VMs even with SDN.Use in the role.
|
|
||||||
# /vms and /nodes/<node> are required for VM creation and start via API.
|
|
||||||
if [[ "$DETECT_API_ACLS" != "yes" ]]; then
|
|
||||||
remote "pveum acl modify /vms -user automation@pve -role IncusOSDeployer"
|
|
||||||
remote "pveum acl modify /nodes/${DETECT_HOSTNAME} -user automation@pve -role IncusOSDeployer"
|
|
||||||
remote "pveum acl modify /pool/IncusLab -user automation@pve -role IncusOSDeployer"
|
remote "pveum acl modify /pool/IncusLab -user automation@pve -role IncusOSDeployer"
|
||||||
remote "pveum acl modify /sdn -user automation@pve -role IncusOSDeployer"
|
|
||||||
|
|
||||||
local storage_name
|
local storage_name
|
||||||
storage_name=$(remote_output "pvesm status 2>/dev/null | awk '/zfspool/{print \$1}' | head -1" || true)
|
storage_name=$(remote_output "pvesm status 2>/dev/null | awk '/zfspool/{print \$1}' | head -1" || true)
|
||||||
|
|
@ -1290,7 +1245,12 @@ apply_api_token() {
|
||||||
fi
|
fi
|
||||||
remote "pveum acl modify /storage/local -user automation@pve -role IncusOSDeployer"
|
remote "pveum acl modify /storage/local -user automation@pve -role IncusOSDeployer"
|
||||||
|
|
||||||
success "ACLs configured"
|
success "API token created"
|
||||||
|
echo ""
|
||||||
|
warn "Token secret (save now -- shown only once):"
|
||||||
|
echo ""
|
||||||
|
echo " ${token_secret}"
|
||||||
|
echo ""
|
||||||
fi
|
fi
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -1302,7 +1262,7 @@ show_summary() {
|
||||||
step "Summary"
|
step "Summary"
|
||||||
|
|
||||||
echo ""
|
echo ""
|
||||||
echo -e "${BOLD}proxmox.yaml for envs/hetzner/:${RESET}"
|
echo -e "${BOLD}proxmox.yaml for incusos/targets/hetzner/:${RESET}"
|
||||||
echo ""
|
echo ""
|
||||||
echo " host: ${SUBNET_GW}"
|
echo " host: ${SUBNET_GW}"
|
||||||
echo " method: api"
|
echo " method: api"
|
||||||
|
|
@ -1321,17 +1281,18 @@ show_summary() {
|
||||||
echo ""
|
echo ""
|
||||||
echo " # Hetzner"
|
echo " # Hetzner"
|
||||||
echo " HETZNER_PROXMOX_TOKEN_SECRET=<token from step above>"
|
echo " HETZNER_PROXMOX_TOKEN_SECRET=<token from step above>"
|
||||||
|
echo " HETZNER_PROXMOX_ROOT_PASSWORD=<your root password>"
|
||||||
|
|
||||||
echo ""
|
echo ""
|
||||||
echo -e "${BOLD}Next steps:${RESET}"
|
echo -e "${BOLD}Next steps:${RESET}"
|
||||||
echo ""
|
echo ""
|
||||||
echo " 1. Save WireGuard client config and connect"
|
echo " 1. Save WireGuard client config and connect"
|
||||||
echo " 2. Copy proxmox.yaml.example and fill in credentials:"
|
echo " 2. Copy proxmox.yaml.example and fill in credentials:"
|
||||||
echo " cp envs/hetzner/proxmox.yaml.example envs/hetzner/proxmox.yaml"
|
echo " cp incusos/targets/hetzner/proxmox.yaml.example incusos/targets/hetzner/proxmox.yaml"
|
||||||
echo " 3. Test with dry run:"
|
echo " 3. Test with dry run:"
|
||||||
echo " export PROXMOX_TOKEN_SECRET=\"\$HETZNER_PROXMOX_TOKEN_SECRET\""
|
echo " export PROXMOX_TOKEN_SECRET=\"\$HETZNER_PROXMOX_TOKEN_SECRET\""
|
||||||
echo " bin/incusos-proxmox --dry-run --proxmox envs/hetzner/proxmox.yaml \\"
|
echo " incusos-proxmox --dry-run --proxmox incusos/targets/hetzner/proxmox.yaml \\"
|
||||||
echo " envs/hetzner/lab-cluster.yaml"
|
echo " incusos/targets/hetzner/lab-cluster.yaml"
|
||||||
echo ""
|
echo ""
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -1371,7 +1332,7 @@ main() {
|
||||||
apply_repos
|
apply_repos
|
||||||
apply_update
|
apply_update
|
||||||
apply_bridge
|
apply_bridge
|
||||||
apply_dnsmasq
|
apply_storage
|
||||||
apply_wireguard
|
apply_wireguard
|
||||||
apply_firewall
|
apply_firewall
|
||||||
apply_api_token
|
apply_api_token
|
||||||
|
|
@ -376,7 +376,7 @@ directory and current directory. Override with `--proxmox FILE`.
|
||||||
Copy the template to get started:
|
Copy the template to get started:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cp bin/examples/proxmox.yaml.example envs/beelink/proxmox.yaml
|
cp incusos/examples/proxmox.yaml.example incusos/proxmox.yaml
|
||||||
# Edit with your Proxmox host, credentials, and storage settings
|
# Edit with your Proxmox host, credentials, and storage settings
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
@ -557,25 +557,25 @@ default — prints what it will do, confirms, executes, and reports pass/fail.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Run all phases (deploy, single-node tests, cluster, workloads, migration)
|
# Run all phases (deploy, single-node tests, cluster, workloads, migration)
|
||||||
bin/lab-test examples/lab-cluster.yaml
|
./incusos/lab-test examples/lab-cluster.yaml
|
||||||
|
|
||||||
# Deploy and verify only
|
# Deploy and verify only
|
||||||
bin/lab-test --phase deploy examples/lab-cluster.yaml
|
./incusos/lab-test --phase deploy examples/lab-cluster.yaml
|
||||||
|
|
||||||
# Single-node workload tests
|
# Single-node workload tests
|
||||||
bin/lab-test --phase single examples/lab-cluster.yaml
|
./incusos/lab-test --phase single examples/lab-cluster.yaml
|
||||||
|
|
||||||
# Form cluster from deployed nodes
|
# Form cluster from deployed nodes
|
||||||
bin/lab-test --phase cluster examples/lab-cluster.yaml
|
./incusos/lab-test --phase cluster examples/lab-cluster.yaml
|
||||||
|
|
||||||
# Test migration and evacuation
|
# Test migration and evacuation
|
||||||
bin/lab-test --phase migrate examples/lab-cluster.yaml
|
./incusos/lab-test --phase migrate examples/lab-cluster.yaml
|
||||||
|
|
||||||
# Full run without prompts
|
# Full run without prompts
|
||||||
bin/lab-test --yes examples/lab-cluster.yaml
|
./incusos/lab-test --yes examples/lab-cluster.yaml
|
||||||
|
|
||||||
# Clean up test instances (keeps VMs)
|
# Clean up test instances (keeps VMs)
|
||||||
bin/lab-test --cleanup examples/lab-cluster.yaml
|
./incusos/lab-test --cleanup examples/lab-cluster.yaml
|
||||||
```
|
```
|
||||||
|
|
||||||
### Phases
|
### Phases
|
||||||
|
|
@ -140,16 +140,17 @@ key auth first (`ssh-copy-id root@<host>`).
|
||||||
### Step 2: Set Up Proxmox Connection
|
### Step 2: Set Up Proxmox Connection
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cp bin/examples/proxmox.yaml.example envs/beelink/proxmox.yaml
|
cp incusos/examples/proxmox.yaml.example incusos/proxmox.yaml
|
||||||
```
|
```
|
||||||
|
|
||||||
Edit `envs/beelink/proxmox.yaml` with your Proxmox host settings, or
|
Edit `incusos/proxmox.yaml` with your Proxmox host settings. This is
|
||||||
use `--proxmox envs/beelink/proxmox.yaml` to point at it explicitly.
|
auto-discovered by the scripts and avoids repeating credentials in each
|
||||||
|
lab config.
|
||||||
|
|
||||||
### Step 3: Create a Test Config
|
### Step 3: Create a Test Config
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cp bin/examples/proxmox-minimal.yaml my-test.yaml
|
cp incusos/examples/proxmox-minimal.yaml my-test.yaml
|
||||||
```
|
```
|
||||||
|
|
||||||
Edit `my-test.yaml` -- change only the host IP (or rely on proxmox.yaml):
|
Edit `my-test.yaml` -- change only the host IP (or rely on proxmox.yaml):
|
||||||
|
|
@ -175,7 +176,7 @@ Using VMID 900+ avoids clashing with existing VMs.
|
||||||
### Step 4: Dry Run
|
### Step 4: Dry Run
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
bin/incusos-proxmox --dry-run my-test.yaml
|
./incusos/incusos-proxmox --dry-run my-test.yaml
|
||||||
```
|
```
|
||||||
|
|
||||||
**Verify in the output:**
|
**Verify in the output:**
|
||||||
|
|
@ -192,7 +193,7 @@ bin/incusos-proxmox --dry-run my-test.yaml
|
||||||
### Step 5: Deploy (Single VM)
|
### Step 5: Deploy (Single VM)
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
bin/incusos-proxmox my-test.yaml
|
./incusos/incusos-proxmox my-test.yaml
|
||||||
```
|
```
|
||||||
|
|
||||||
**Watch the phases:**
|
**Watch the phases:**
|
||||||
|
|
@ -218,7 +219,7 @@ bin/incusos-proxmox my-test.yaml
|
||||||
After the deploy completes, verify with the built-in status command:
|
After the deploy completes, verify with the built-in status command:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
bin/incusos-proxmox --status my-test.yaml
|
./incusos/incusos-proxmox --status my-test.yaml
|
||||||
```
|
```
|
||||||
|
|
||||||
**Verify in the output:**
|
**Verify in the output:**
|
||||||
|
|
@ -255,7 +256,7 @@ check the Proxmox console — stale ARP entries can be misleading.
|
||||||
Re-run the deploy to verify the reconcile menu:
|
Re-run the deploy to verify the reconcile menu:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
bin/incusos-proxmox my-test.yaml
|
./incusos/incusos-proxmox my-test.yaml
|
||||||
```
|
```
|
||||||
|
|
||||||
**Verify:** The script detects the existing VM and shows the interactive menu.
|
**Verify:** The script detects the existing VM and shows the interactive menu.
|
||||||
|
|
@ -264,7 +265,7 @@ Choose option 1 to run status checks without modifying anything.
|
||||||
Also test with `--yes` to verify it defaults to safe behavior:
|
Also test with `--yes` to verify it defaults to safe behavior:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
bin/incusos-proxmox --yes my-test.yaml
|
./incusos/incusos-proxmox --yes my-test.yaml
|
||||||
```
|
```
|
||||||
|
|
||||||
**Verify:** Runs post-deployment checks, does NOT destroy.
|
**Verify:** Runs post-deployment checks, does NOT destroy.
|
||||||
|
|
@ -272,7 +273,7 @@ bin/incusos-proxmox --yes my-test.yaml
|
||||||
### Step 9: Clean Up the Test VM
|
### Step 9: Clean Up the Test VM
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
bin/incusos-proxmox --cleanup my-test.yaml
|
./incusos/incusos-proxmox --cleanup my-test.yaml
|
||||||
```
|
```
|
||||||
|
|
||||||
- Confirms before destroying
|
- Confirms before destroying
|
||||||
|
|
@ -345,9 +346,9 @@ proxmox:
|
||||||
Deploy and verify:
|
Deploy and verify:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
bin/incusos-proxmox --dry-run my-test.yaml # Should show pool in plan
|
./incusos/incusos-proxmox --dry-run my-test.yaml # Should show pool in plan
|
||||||
bin/incusos-proxmox my-test.yaml # VM created in pool
|
./incusos/incusos-proxmox my-test.yaml # VM created in pool
|
||||||
bin/incusos-proxmox --status my-test.yaml # Status scoped to pool
|
./incusos/incusos-proxmox --status my-test.yaml # Status scoped to pool
|
||||||
```
|
```
|
||||||
|
|
||||||
### Step 12: (Optional) Full Lab Deployment
|
### Step 12: (Optional) Full Lab Deployment
|
||||||
|
|
@ -355,10 +356,10 @@ bin/incusos-proxmox --status my-test.yaml # Status scoped to pool
|
||||||
Once single-VM tests pass, deploy the full 4-VM lab:
|
Once single-VM tests pass, deploy the full 4-VM lab:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cp bin/examples/proxmox-lab.yaml my-lab.yaml
|
cp incusos/examples/proxmox-lab.yaml my-lab.yaml
|
||||||
# Edit: change host IP, adjust start_vmid if needed
|
# Edit: change host IP, adjust start_vmid if needed
|
||||||
bin/incusos-proxmox --dry-run my-lab.yaml
|
./incusos/incusos-proxmox --dry-run my-lab.yaml
|
||||||
bin/incusos-proxmox my-lab.yaml
|
./incusos/incusos-proxmox my-lab.yaml
|
||||||
```
|
```
|
||||||
|
|
||||||
This deploys 1 Operations Center + 3 Incus nodes. After deployment:
|
This deploys 1 Operations Center + 3 Incus nodes. After deployment:
|
||||||
|
|
@ -378,44 +379,44 @@ VMs: single-node workloads, cluster formation, migration, and evacuation.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# 1. Check environment
|
# 1. Check environment
|
||||||
bin/incusos-proxmox --doctor
|
./incusos/incusos-proxmox --doctor
|
||||||
|
|
||||||
# 2. Deploy 3 nodes
|
# 2. Deploy 3 nodes
|
||||||
bin/incusos-proxmox --yes examples/lab-cluster.yaml
|
./incusos/incusos-proxmox --yes examples/lab-cluster.yaml
|
||||||
|
|
||||||
# 3. Add incus remotes for each node (use IPs from --status)
|
# 3. Add incus remotes for each node (use IPs from --status)
|
||||||
bin/incusos-proxmox --status examples/lab-cluster.yaml
|
./incusos/incusos-proxmox --status examples/lab-cluster.yaml
|
||||||
incus remote add incus-lab-01 <IP1> --accept-certificate
|
incus remote add incus-lab-01 <IP1> --accept-certificate
|
||||||
incus remote add incus-lab-02 <IP2> --accept-certificate
|
incus remote add incus-lab-02 <IP2> --accept-certificate
|
||||||
incus remote add incus-lab-03 <IP3> --accept-certificate
|
incus remote add incus-lab-03 <IP3> --accept-certificate
|
||||||
|
|
||||||
# 4. Run full lab validation
|
# 4. Run full lab validation
|
||||||
bin/lab-test --yes examples/lab-cluster.yaml
|
./incusos/lab-test --yes examples/lab-cluster.yaml
|
||||||
```
|
```
|
||||||
|
|
||||||
### Phase-by-Phase Testing
|
### Phase-by-Phase Testing
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Single-node workloads only (container + VM launch/exec)
|
# Single-node workloads only (container + VM launch/exec)
|
||||||
bin/lab-test --phase single --skip-deploy examples/lab-cluster.yaml
|
./incusos/lab-test --phase single --skip-deploy examples/lab-cluster.yaml
|
||||||
|
|
||||||
# Cluster formation only
|
# Cluster formation only
|
||||||
bin/lab-test --phase cluster --skip-deploy examples/lab-cluster.yaml
|
./incusos/lab-test --phase cluster --skip-deploy examples/lab-cluster.yaml
|
||||||
|
|
||||||
# Cluster workloads (scheduler placement + targeted placement)
|
# Cluster workloads (scheduler placement + targeted placement)
|
||||||
bin/lab-test --phase workload --skip-deploy examples/lab-cluster.yaml
|
./incusos/lab-test --phase workload --skip-deploy examples/lab-cluster.yaml
|
||||||
|
|
||||||
# Migration and evacuation tests
|
# Migration and evacuation tests
|
||||||
bin/lab-test --phase migrate --skip-deploy examples/lab-cluster.yaml
|
./incusos/lab-test --phase migrate --skip-deploy examples/lab-cluster.yaml
|
||||||
|
|
||||||
# Extended phases (require a working cluster):
|
# Extended phases (require a working cluster):
|
||||||
bin/lab-test --phase storage --skip-deploy examples/lab-cluster.yaml
|
./incusos/lab-test --phase storage --skip-deploy examples/lab-cluster.yaml
|
||||||
bin/lab-test --phase network --skip-deploy examples/lab-cluster.yaml
|
./incusos/lab-test --phase network --skip-deploy examples/lab-cluster.yaml
|
||||||
bin/lab-test --phase projects --skip-deploy examples/lab-cluster.yaml
|
./incusos/lab-test --phase projects --skip-deploy examples/lab-cluster.yaml
|
||||||
bin/lab-test --phase backup --skip-deploy examples/lab-cluster.yaml
|
./incusos/lab-test --phase backup --skip-deploy examples/lab-cluster.yaml
|
||||||
bin/lab-test --phase limits --skip-deploy examples/lab-cluster.yaml
|
./incusos/lab-test --phase limits --skip-deploy examples/lab-cluster.yaml
|
||||||
bin/lab-test --phase security --skip-deploy examples/lab-cluster.yaml
|
./incusos/lab-test --phase security --skip-deploy examples/lab-cluster.yaml
|
||||||
bin/lab-test --phase resilience --skip-deploy examples/lab-cluster.yaml
|
./incusos/lab-test --phase resilience --skip-deploy examples/lab-cluster.yaml
|
||||||
```
|
```
|
||||||
|
|
||||||
### Extended Test Phases
|
### Extended Test Phases
|
||||||
|
|
@ -659,17 +660,17 @@ and detailed test results.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Remove test instances only (keeps VMs and cluster)
|
# Remove test instances only (keeps VMs and cluster)
|
||||||
bin/lab-test --cleanup examples/lab-cluster.yaml
|
./incusos/lab-test --cleanup examples/lab-cluster.yaml
|
||||||
|
|
||||||
# Destroy VMs only
|
# Destroy VMs only
|
||||||
bin/incusos-proxmox --cleanup examples/lab-cluster.yaml
|
./incusos/incusos-proxmox --cleanup examples/lab-cluster.yaml
|
||||||
|
|
||||||
# Full cleanup: VMs + deployment ISO + seeds + remotes + cache
|
# Full cleanup: VMs + deployment ISO + seeds + remotes + cache
|
||||||
bin/incusos-proxmox --cleanup --deep --yes examples/lab-cluster.yaml
|
./incusos/incusos-proxmox --cleanup --deep --yes examples/lab-cluster.yaml
|
||||||
|
|
||||||
# Pool-wide: destroy ALL managed VMs (no config file needed)
|
# Pool-wide: destroy ALL managed VMs (no config file needed)
|
||||||
bin/incusos-proxmox --cleanup-all --yes
|
./incusos/incusos-proxmox --cleanup-all --yes
|
||||||
bin/incusos-proxmox --cleanup-all --deep --yes # nuke everything
|
./incusos/incusos-proxmox --cleanup-all --deep --yes # nuke everything
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
@ -706,14 +707,14 @@ openssl pkcs12 -export \
|
||||||
-out ~/.config/incus/client.pfx
|
-out ~/.config/incus/client.pfx
|
||||||
|
|
||||||
# 4. Verify
|
# 4. Verify
|
||||||
bin/incusos-proxmox --doctor
|
./incusos/incusos-proxmox --doctor
|
||||||
```
|
```
|
||||||
|
|
||||||
### Step 1: Deploy OC Server
|
### Step 1: Deploy OC Server
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
bin/incusos-proxmox --yes bin/examples/lab-oc-deploy.yaml
|
./incusos/incusos-proxmox --yes incusos/examples/lab-oc-deploy.yaml
|
||||||
bin/incusos-proxmox --status bin/examples/lab-oc-deploy.yaml
|
./incusos/incusos-proxmox --status incusos/examples/lab-oc-deploy.yaml
|
||||||
# Note OC_IP from output
|
# Note OC_IP from output
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
@ -762,9 +763,9 @@ ls -lh /tmp/IncusOS-oc.iso # ~3.4 GB
|
||||||
### Step 6: Deploy Nodes (Hybrid)
|
### Step 6: Deploy Nodes (Hybrid)
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
bin/incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes \
|
./incusos/incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes \
|
||||||
bin/examples/lab-oc-nodes.yaml
|
incusos/examples/lab-oc-nodes.yaml
|
||||||
bin/incusos-proxmox --status bin/examples/lab-oc-nodes.yaml
|
./incusos/incusos-proxmox --status incusos/examples/lab-oc-nodes.yaml
|
||||||
```
|
```
|
||||||
|
|
||||||
`incusos-proxmox` creates VMs, uploads the OC ISO to ide2, generates per-node
|
`incusos-proxmox` creates VMs, uploads the OC ISO to ide2, generates per-node
|
||||||
|
|
@ -857,10 +858,10 @@ incus delete oc-node-01:test-ct --force 2>/dev/null
|
||||||
incus delete oc-node-01:test-vm --force 2>/dev/null
|
incus delete oc-node-01:test-vm --force 2>/dev/null
|
||||||
|
|
||||||
# Nodes
|
# Nodes
|
||||||
bin/incusos-proxmox --cleanup --deep --yes bin/examples/lab-oc-nodes.yaml
|
./incusos/incusos-proxmox --cleanup --deep --yes incusos/examples/lab-oc-nodes.yaml
|
||||||
|
|
||||||
# OC server
|
# OC server
|
||||||
bin/incusos-proxmox --cleanup --deep --yes bin/examples/lab-oc-deploy.yaml
|
./incusos/incusos-proxmox --cleanup --deep --yes incusos/examples/lab-oc-deploy.yaml
|
||||||
|
|
||||||
# Remotes
|
# Remotes
|
||||||
operations-center remote remove oc-lab 2>/dev/null
|
operations-center remote remove oc-lab 2>/dev/null
|
||||||
|
|
@ -13,7 +13,7 @@ set -euo pipefail
|
||||||
readonly VERSION="0.1.0"
|
readonly VERSION="0.1.0"
|
||||||
readonly SCRIPT_NAME="deploy-awx"
|
readonly SCRIPT_NAME="deploy-awx"
|
||||||
readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
readonly MANIFESTS_DIR="${SCRIPT_DIR}/data/awx-manifests"
|
readonly MANIFESTS_DIR="${SCRIPT_DIR}/awx-manifests"
|
||||||
readonly AWX_NAMESPACE="awx"
|
readonly AWX_NAMESPACE="awx"
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
@ -21,26 +21,24 @@ readonly AWX_NAMESPACE="awx"
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
VM_NAME="awx"
|
VM_NAME="awx"
|
||||||
TARGET_REMOTE=""
|
TARGET_REMOTE="oc-node-02"
|
||||||
TARGET_NODE="" # Node within cluster for --target; defaults to TARGET_REMOTE if empty
|
AWX_IP="192.168.102.161"
|
||||||
AWX_IP=""
|
|
||||||
AWX_PREFIX="22"
|
AWX_PREFIX="22"
|
||||||
AWX_GATEWAY=""
|
AWX_GATEWAY="192.168.100.1"
|
||||||
AWX_DNS=""
|
AWX_DNS="192.168.100.1"
|
||||||
AWX_CPU="4"
|
AWX_CPU="4"
|
||||||
AWX_MEMORY="8GiB"
|
AWX_MEMORY="8GiB"
|
||||||
AWX_DISK="40GiB"
|
AWX_DISK="40GiB"
|
||||||
GIT_REPO=""
|
GIT_REPO="ssh://git@192.168.1.200:2222/maarten/incus-contrib.git"
|
||||||
GIT_BRANCH="master"
|
GIT_BRANCH="master"
|
||||||
PLAYBOOK_DIR="playbooks"
|
PLAYBOOK_DIR="playbooks"
|
||||||
AETHER_URL=""
|
AETHER_URL="https://192.168.102.160:8443"
|
||||||
AETHER_CLUSTER_ID=""
|
AETHER_CLUSTER_ID="52"
|
||||||
|
|
||||||
# Runtime flags
|
# Runtime flags
|
||||||
DRY_RUN=false
|
DRY_RUN=false
|
||||||
VERBOSE=false
|
VERBOSE=false
|
||||||
QUIET=false
|
QUIET=false
|
||||||
RESUME=false
|
|
||||||
CONFIG_FILE=""
|
CONFIG_FILE=""
|
||||||
ACTION=""
|
ACTION=""
|
||||||
|
|
||||||
|
|
@ -165,10 +163,8 @@ parse_config() {
|
||||||
local val
|
local val
|
||||||
val=$(grep -A50 '^awx:' "$file" | grep '^ *vm_name:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
|
val=$(grep -A50 '^awx:' "$file" | grep '^ *vm_name:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
|
||||||
[[ -n "$val" ]] && VM_NAME="$val"
|
[[ -n "$val" ]] && VM_NAME="$val"
|
||||||
val=$(grep -A50 '^awx:' "$file" | grep '^ *target_remote:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
|
|
||||||
[[ -n "$val" ]] && TARGET_REMOTE="$val"
|
|
||||||
val=$(grep -A50 '^awx:' "$file" | grep '^ *target_node:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
|
val=$(grep -A50 '^awx:' "$file" | grep '^ *target_node:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
|
||||||
[[ -n "$val" ]] && TARGET_NODE="$val"
|
[[ -n "$val" ]] && TARGET_REMOTE="$val"
|
||||||
val=$(grep -A50 '^awx:' "$file" | grep '^ *ip:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
|
val=$(grep -A50 '^awx:' "$file" | grep '^ *ip:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
|
||||||
if [[ -n "$val" ]]; then
|
if [[ -n "$val" ]]; then
|
||||||
AWX_IP="${val%%/*}"
|
AWX_IP="${val%%/*}"
|
||||||
|
|
@ -188,7 +184,7 @@ parse_config() {
|
||||||
[[ -n "$val" ]] && GIT_REPO="$val"
|
[[ -n "$val" ]] && GIT_REPO="$val"
|
||||||
val=$(grep -A50 '^awx:' "$file" | grep '^ *git_branch:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
|
val=$(grep -A50 '^awx:' "$file" | grep '^ *git_branch:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
|
||||||
[[ -n "$val" ]] && GIT_BRANCH="$val"
|
[[ -n "$val" ]] && GIT_BRANCH="$val"
|
||||||
val=$(grep -A50 '^awx:' "$file" | grep '^ *aether_url:' | head -1 | sed 's/^ *aether_url: *//' | tr -d '"' || true)
|
val=$(grep -A50 '^awx:' "$file" | grep '^ *aether_url:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
|
||||||
[[ -n "$val" ]] && AETHER_URL="$val"
|
[[ -n "$val" ]] && AETHER_URL="$val"
|
||||||
val=$(grep -A50 '^awx:' "$file" | grep '^ *aether_cluster_id:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
|
val=$(grep -A50 '^awx:' "$file" | grep '^ *aether_cluster_id:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
|
||||||
[[ -n "$val" ]] && AETHER_CLUSTER_ID="$val"
|
[[ -n "$val" ]] && AETHER_CLUSTER_ID="$val"
|
||||||
|
|
@ -379,10 +375,8 @@ action_deploy() {
|
||||||
echo
|
echo
|
||||||
|
|
||||||
if vm_exists; then
|
if vm_exists; then
|
||||||
if [[ "${RESUME:-false}" == true ]]; then
|
if vm_running; then
|
||||||
info "AWX VM already exists — resuming from phase 4"
|
error "AWX VM already exists and is running. Use --status, --heal, or --cleanup first."
|
||||||
elif vm_running; then
|
|
||||||
error "AWX VM already exists and is running. Use --status, --heal, --resume, or --cleanup first."
|
|
||||||
return 1
|
return 1
|
||||||
else
|
else
|
||||||
warn "AWX VM exists but is not running — starting it"
|
warn "AWX VM exists but is not running — starting it"
|
||||||
|
|
@ -393,11 +387,9 @@ action_deploy() {
|
||||||
fi
|
fi
|
||||||
fi
|
fi
|
||||||
|
|
||||||
if [[ "${RESUME:-false}" != true ]]; then
|
phase_create_vm
|
||||||
phase_create_vm
|
phase_configure_network
|
||||||
phase_configure_network
|
phase_install_k3s
|
||||||
phase_install_k3s
|
|
||||||
fi
|
|
||||||
phase_deploy_awx_operator
|
phase_deploy_awx_operator
|
||||||
phase_deploy_awx_instance
|
phase_deploy_awx_instance
|
||||||
phase_verify
|
phase_verify
|
||||||
|
|
@ -418,13 +410,12 @@ action_deploy() {
|
||||||
phase_create_vm() {
|
phase_create_vm() {
|
||||||
step "Phase 1: Create VM"
|
step "Phase 1: Create VM"
|
||||||
|
|
||||||
local target_node="${TARGET_NODE:-$TARGET_REMOTE}"
|
if ! dry_run_guard "incus launch images:debian/12 $(vm_ref) --vm --target ${TARGET_REMOTE} ..."; then
|
||||||
if ! dry_run_guard "incus launch images:debian/12 $(vm_ref) --vm --target ${target_node} ..."; then
|
|
||||||
return 0
|
return 0
|
||||||
fi
|
fi
|
||||||
|
|
||||||
incus launch images:debian/12 "$(vm_ref)" --vm \
|
incus launch images:debian/12 "$(vm_ref)" --vm \
|
||||||
--target "$target_node" \
|
--target "$TARGET_REMOTE" \
|
||||||
-c "limits.cpu=${AWX_CPU}" \
|
-c "limits.cpu=${AWX_CPU}" \
|
||||||
-c "limits.memory=${AWX_MEMORY}" \
|
-c "limits.memory=${AWX_MEMORY}" \
|
||||||
-d "root,size=${AWX_DISK}"
|
-d "root,size=${AWX_DISK}"
|
||||||
|
|
@ -499,7 +490,7 @@ phase_install_k3s() {
|
||||||
vm_exec bash -c "
|
vm_exec bash -c "
|
||||||
export DEBIAN_FRONTEND=noninteractive
|
export DEBIAN_FRONTEND=noninteractive
|
||||||
apt-get update -qq
|
apt-get update -qq
|
||||||
apt-get install -y -qq curl python3 git 2>&1 | tail -1
|
apt-get install -y -qq curl python3 2>&1 | tail -1
|
||||||
"
|
"
|
||||||
|
|
||||||
vm_exec bash -c "
|
vm_exec bash -c "
|
||||||
|
|
@ -1171,7 +1162,6 @@ parse_args() {
|
||||||
--cleanup) ACTION="cleanup";;
|
--cleanup) ACTION="cleanup";;
|
||||||
--doctor) ACTION="doctor";;
|
--doctor) ACTION="doctor";;
|
||||||
-c|--config) CONFIG_FILE="$2"; shift;;
|
-c|--config) CONFIG_FILE="$2"; shift;;
|
||||||
--resume) RESUME=true;;
|
|
||||||
-n|--dry-run) DRY_RUN=true;;
|
-n|--dry-run) DRY_RUN=true;;
|
||||||
-v|--verbose) VERBOSE=true;;
|
-v|--verbose) VERBOSE=true;;
|
||||||
-q|--quiet) QUIET=true;;
|
-q|--quiet) QUIET=true;;
|
||||||
|
|
@ -1196,21 +1186,6 @@ parse_args() {
|
||||||
# Main
|
# Main
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
require_config() {
|
|
||||||
local missing=()
|
|
||||||
[[ -z "$TARGET_REMOTE" ]] && missing+=("target_remote")
|
|
||||||
[[ -z "$AWX_IP" ]] && missing+=("ip")
|
|
||||||
[[ -z "$AWX_GATEWAY" ]] && missing+=("gateway")
|
|
||||||
[[ -z "$AWX_DNS" ]] && missing+=("dns")
|
|
||||||
[[ -z "$AETHER_URL" ]] && missing+=("aether_url")
|
|
||||||
[[ -z "$AETHER_CLUSTER_ID" ]] && missing+=("aether_cluster_id")
|
|
||||||
if [[ ${#missing[@]} -gt 0 ]]; then
|
|
||||||
error "Required config missing: ${missing[*]}"
|
|
||||||
error "Use --config FILE to load environment settings (see envs/)"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
main() {
|
main() {
|
||||||
setup_colors
|
setup_colors
|
||||||
parse_args "$@"
|
parse_args "$@"
|
||||||
|
|
@ -1220,11 +1195,6 @@ main() {
|
||||||
parse_config "$CONFIG_FILE"
|
parse_config "$CONFIG_FILE"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
# Validate required fields (must come from --config or CLI flags)
|
|
||||||
if [[ "$ACTION" != "doctor" ]]; then
|
|
||||||
require_config
|
|
||||||
fi
|
|
||||||
|
|
||||||
echo
|
echo
|
||||||
case "$ACTION" in
|
case "$ACTION" in
|
||||||
deploy) action_deploy;;
|
deploy) action_deploy;;
|
||||||
|
|
@ -16,26 +16,27 @@ set -euo pipefail
|
||||||
readonly VERSION="0.1.0"
|
readonly VERSION="0.1.0"
|
||||||
readonly SCRIPT_NAME="deploy-haproxy"
|
readonly SCRIPT_NAME="deploy-haproxy"
|
||||||
readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
readonly ENV_FILE="${SCRIPT_DIR}/../env"
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Defaults (overridden by config file or CLI flags)
|
# Defaults (overridden by config file or CLI flags)
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
CLUSTER_REMOTE=""
|
CLUSTER_REMOTE="oc-node-01"
|
||||||
CLUSTER_ID=""
|
CLUSTER_ID="52"
|
||||||
OVN_NETWORK="net-prod"
|
OVN_NETWORK="net-prod"
|
||||||
VIP=""
|
VIP="192.168.103.200"
|
||||||
HAPROXY_01_IP=""
|
HAPROXY_01_IP="10.10.10.50"
|
||||||
HAPROXY_02_IP=""
|
HAPROXY_02_IP="10.10.10.51"
|
||||||
HAPROXY_CPU="2"
|
HAPROXY_CPU="2"
|
||||||
HAPROXY_MEMORY="1GB"
|
HAPROXY_MEMORY="1GB"
|
||||||
BACKEND_IPS=""
|
BACKEND_IPS="10.10.10.60,10.10.10.61,10.10.10.62"
|
||||||
BACKEND_PORT="80"
|
BACKEND_PORT="80"
|
||||||
SERVICE_NAME="web-test"
|
SERVICE_NAME="web-test"
|
||||||
SERVICE_MODE="http"
|
SERVICE_MODE="http"
|
||||||
SERVICE_BALANCE="roundrobin"
|
SERVICE_BALANCE="roundrobin"
|
||||||
IMAGE_VERSION="1.0.0"
|
IMAGE_VERSION="1.0.0"
|
||||||
AETHER_URL=""
|
AETHER_URL="https://192.168.102.160:8443"
|
||||||
AETHER_USER="admin"
|
AETHER_USER="admin"
|
||||||
AETHER_PASSWORD=""
|
AETHER_PASSWORD=""
|
||||||
|
|
||||||
|
|
@ -183,7 +184,7 @@ parse_config() {
|
||||||
[[ -n "$val" ]] && SERVICE_NAME="$val"
|
[[ -n "$val" ]] && SERVICE_NAME="$val"
|
||||||
val=$(grep -A50 '^haproxy:' "$file" | grep '^ *image_version:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
|
val=$(grep -A50 '^haproxy:' "$file" | grep '^ *image_version:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
|
||||||
[[ -n "$val" ]] && IMAGE_VERSION="$val"
|
[[ -n "$val" ]] && IMAGE_VERSION="$val"
|
||||||
val=$(grep -A50 '^haproxy:' "$file" | grep '^ *aether_url:' | head -1 | sed 's/^ *aether_url: *//' | tr -d '"' || true)
|
val=$(grep -A50 '^haproxy:' "$file" | grep '^ *aether_url:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
|
||||||
[[ -n "$val" ]] && AETHER_URL="$val"
|
[[ -n "$val" ]] && AETHER_URL="$val"
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -196,29 +197,18 @@ load_password() {
|
||||||
return 0
|
return 0
|
||||||
fi
|
fi
|
||||||
|
|
||||||
# Check if already exported (user sourced envs/*/env)
|
if [[ -f "$ENV_FILE" ]]; then
|
||||||
if [[ -n "${AETHER_ADMIN_PASSWORD:-}" ]]; then
|
local val
|
||||||
AETHER_PASSWORD="$AETHER_ADMIN_PASSWORD"
|
val=$(grep 'AETHER_ADMIN_PASSWORD=' "$ENV_FILE" | head -1 | sed 's/^.*AETHER_ADMIN_PASSWORD=//' | tr -d '"' | tr -d "'" || true)
|
||||||
return 0
|
if [[ -n "$val" ]]; then
|
||||||
|
AETHER_PASSWORD="$val"
|
||||||
|
detail "Password loaded from env file"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
fi
|
fi
|
||||||
|
|
||||||
# Search for env file: walk up from script dir
|
|
||||||
local dir="$SCRIPT_DIR"
|
|
||||||
while [[ "$dir" != "/" ]]; do
|
|
||||||
if [[ -f "${dir}/env" ]]; then
|
|
||||||
local val
|
|
||||||
val=$(grep 'AETHER_ADMIN_PASSWORD=' "${dir}/env" | head -1 | sed 's/^.*AETHER_ADMIN_PASSWORD=//' | tr -d '"' | tr -d "'" || true)
|
|
||||||
if [[ -n "$val" ]]; then
|
|
||||||
AETHER_PASSWORD="$val"
|
|
||||||
detail "Password loaded from ${dir}/env"
|
|
||||||
return 0
|
|
||||||
fi
|
|
||||||
fi
|
|
||||||
dir="$(dirname "$dir")"
|
|
||||||
done
|
|
||||||
|
|
||||||
error "Aether password not found"
|
error "Aether password not found"
|
||||||
error "Source envs/<env>/env or use --password"
|
error "Set AETHER_ADMIN_PASSWORD in ${ENV_FILE} or use --password"
|
||||||
return 1
|
return 1
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -1200,22 +1190,6 @@ parse_args() {
|
||||||
# Main
|
# Main
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
require_config() {
|
|
||||||
local missing=()
|
|
||||||
[[ -z "$CLUSTER_REMOTE" ]] && missing+=("cluster_remote")
|
|
||||||
[[ -z "$CLUSTER_ID" ]] && missing+=("cluster_id")
|
|
||||||
[[ -z "$VIP" ]] && missing+=("vip")
|
|
||||||
[[ -z "$HAPROXY_01_IP" ]] && missing+=("haproxy_01_ip")
|
|
||||||
[[ -z "$HAPROXY_02_IP" ]] && missing+=("haproxy_02_ip")
|
|
||||||
[[ -z "$BACKEND_IPS" ]] && missing+=("backend_ips")
|
|
||||||
[[ -z "$AETHER_URL" ]] && missing+=("aether_url")
|
|
||||||
if [[ ${#missing[@]} -gt 0 ]]; then
|
|
||||||
error "Required config missing: ${missing[*]}"
|
|
||||||
error "Use --config FILE to load environment settings (see envs/)"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
main() {
|
main() {
|
||||||
setup_colors
|
setup_colors
|
||||||
parse_args "$@"
|
parse_args "$@"
|
||||||
|
|
@ -1225,11 +1199,6 @@ main() {
|
||||||
parse_config "$CONFIG_FILE"
|
parse_config "$CONFIG_FILE"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
# Validate required fields (must come from --config or CLI flags)
|
|
||||||
if [[ "$ACTION" != "doctor" ]]; then
|
|
||||||
require_config
|
|
||||||
fi
|
|
||||||
|
|
||||||
echo
|
echo
|
||||||
case "$ACTION" in
|
case "$ACTION" in
|
||||||
deploy) action_deploy;;
|
deploy) action_deploy;;
|
||||||
|
|
@ -14,37 +14,35 @@ set -euo pipefail
|
||||||
readonly VERSION="0.1.0"
|
readonly VERSION="0.1.0"
|
||||||
readonly SCRIPT_NAME="deploy-observability"
|
readonly SCRIPT_NAME="deploy-observability"
|
||||||
readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
readonly DASHBOARDS_DIR="${SCRIPT_DIR}/data/observability-dashboards"
|
readonly DASHBOARDS_DIR="${SCRIPT_DIR}/observability-dashboards"
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Defaults (overridden by config file or CLI flags)
|
# Defaults (overridden by CLI flags)
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
CLUSTER_REMOTE=""
|
CLUSTER_REMOTE="oc-node-01"
|
||||||
OVN_NETWORK="net-prod"
|
OVN_NETWORK="net-prod"
|
||||||
MONITORING_IP="10.10.10.70"
|
MONITORING_IP="10.10.10.70"
|
||||||
MONITORING_TARGET=""
|
MONITORING_TARGET="oc-node-02"
|
||||||
FORWARD_VIP=""
|
FORWARD_VIP="192.168.103.201"
|
||||||
|
|
||||||
# Fallback defaults — used when cluster is unreachable (e.g., --dry-run)
|
# Fallback defaults — used when cluster is unreachable (e.g., --dry-run)
|
||||||
DEFAULT_INCUS_NODES=()
|
DEFAULT_INCUS_NODES=("192.168.102.140" "192.168.102.141" "192.168.102.142")
|
||||||
DEFAULT_NODE_EXP_TARGETS=()
|
DEFAULT_NODE_EXP_TARGETS=("oc-node-01" "oc-node-02" "oc-node-03")
|
||||||
DEFAULT_NODE_EXP_IPS=()
|
DEFAULT_NODE_EXP_IPS=("10.10.10.71" "10.10.10.72" "10.10.10.73")
|
||||||
|
|
||||||
INCUS_NODES=()
|
INCUS_NODES=()
|
||||||
NODE_EXP_TARGETS=()
|
NODE_EXP_TARGETS=()
|
||||||
NODE_EXP_IPS=()
|
NODE_EXP_IPS=()
|
||||||
HAPROXY_IPS=()
|
HAPROXY_IPS=("10.10.10.50" "10.10.10.51")
|
||||||
|
|
||||||
HAPROXY_CLUSTER_ID=""
|
HAPROXY_CLUSTER_ID="52"
|
||||||
OC_SERVER_IP=""
|
OC_SERVER_IP=""
|
||||||
|
|
||||||
# Runtime flags
|
# Runtime flags
|
||||||
DRY_RUN=false
|
DRY_RUN=false
|
||||||
VERBOSE=false
|
VERBOSE=false
|
||||||
QUIET=false
|
QUIET=false
|
||||||
RESUME=false
|
|
||||||
CONFIG_FILE=""
|
|
||||||
ACTION=""
|
ACTION=""
|
||||||
MANUAL_NODES=""
|
MANUAL_NODES=""
|
||||||
|
|
||||||
|
|
@ -103,13 +101,9 @@ ${BOLD}ACTIONS${RESET}
|
||||||
--doctor Diagnose common issues
|
--doctor Diagnose common issues
|
||||||
|
|
||||||
${BOLD}OPTIONS${RESET}
|
${BOLD}OPTIONS${RESET}
|
||||||
-c, --config FILE Load environment config from YAML file (see envs/)
|
-r, --remote NAME Incus remote (default: ${CLUSTER_REMOTE})
|
||||||
-r, --remote NAME Incus remote (default: ${CLUSTER_REMOTE})
|
--nodes NODE1,NODE2 Manual node list (skip auto-discovery)
|
||||||
--nodes NODE1,NODE2 Manual node list (skip auto-discovery)
|
--oc-server IP Monitor oc-server at this IP (proxy device for 9100)
|
||||||
--monitoring-target NODE Node to deploy monitoring container on (default: ${MONITORING_TARGET})
|
|
||||||
--monitoring-ip IP IP for monitoring container (default: ${MONITORING_IP})
|
|
||||||
--forward-vip IP OVN external IP for Grafana/Prometheus forward (default: ${FORWARD_VIP})
|
|
||||||
--oc-server IP Monitor oc-server at this IP (proxy device for 9100)
|
|
||||||
-n, --dry-run Preview actions without executing
|
-n, --dry-run Preview actions without executing
|
||||||
-v, --verbose Show detailed output
|
-v, --verbose Show detailed output
|
||||||
-q, --quiet Suppress informational output
|
-q, --quiet Suppress informational output
|
||||||
|
|
@ -304,22 +298,16 @@ action_deploy() {
|
||||||
echo
|
echo
|
||||||
|
|
||||||
if container_exists "monitoring"; then
|
if container_exists "monitoring"; then
|
||||||
if [[ "$RESUME" == true ]]; then
|
error "Monitoring container already exists"
|
||||||
info "Monitoring container already exists — resuming from phase 5b"
|
error "Use --status to check health, or --cleanup first."
|
||||||
else
|
return 1
|
||||||
error "Monitoring container already exists"
|
|
||||||
error "Use --status to check health, --cleanup first, or --resume to continue."
|
|
||||||
return 1
|
|
||||||
fi
|
|
||||||
fi
|
fi
|
||||||
|
|
||||||
if [[ "$RESUME" != true ]]; then
|
phase_deploy_monitoring
|
||||||
phase_deploy_monitoring
|
phase_install_prometheus
|
||||||
phase_install_prometheus
|
phase_install_loki
|
||||||
phase_install_loki
|
phase_install_grafana
|
||||||
phase_install_grafana
|
phase_deploy_node_exporters
|
||||||
phase_deploy_node_exporters
|
|
||||||
fi
|
|
||||||
phase_deploy_oc_server_exporter
|
phase_deploy_oc_server_exporter
|
||||||
phase_create_acl
|
phase_create_acl
|
||||||
phase_configure_haproxy_exporter
|
phase_configure_haproxy_exporter
|
||||||
|
|
@ -1706,48 +1694,6 @@ except Exception as e:
|
||||||
fi
|
fi
|
||||||
}
|
}
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Config file parsing
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
parse_config() {
|
|
||||||
local file="$1"
|
|
||||||
if [[ ! -f "$file" ]]; then
|
|
||||||
error "Config file not found: $file"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
local val
|
|
||||||
val=$(grep -A50 '^observability:' "$file" | grep '^ *cluster_remote:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
|
|
||||||
[[ -n "$val" ]] && CLUSTER_REMOTE="$val"
|
|
||||||
val=$(grep -A50 '^observability:' "$file" | grep '^ *ovn_network:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
|
|
||||||
[[ -n "$val" ]] && OVN_NETWORK="$val"
|
|
||||||
val=$(grep -A50 '^observability:' "$file" | grep '^ *monitoring_ip:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
|
|
||||||
[[ -n "$val" ]] && MONITORING_IP="$val"
|
|
||||||
val=$(grep -A50 '^observability:' "$file" | grep '^ *monitoring_target:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
|
|
||||||
[[ -n "$val" ]] && MONITORING_TARGET="$val"
|
|
||||||
val=$(grep -A50 '^observability:' "$file" | grep '^ *forward_vip:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
|
|
||||||
[[ -n "$val" ]] && FORWARD_VIP="$val"
|
|
||||||
val=$(grep -A50 '^observability:' "$file" | grep '^ *incus_nodes:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
|
|
||||||
if [[ -n "$val" ]]; then
|
|
||||||
IFS=',' read -ra DEFAULT_INCUS_NODES <<< "$val"
|
|
||||||
fi
|
|
||||||
val=$(grep -A50 '^observability:' "$file" | grep '^ *node_exp_targets:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
|
|
||||||
if [[ -n "$val" ]]; then
|
|
||||||
IFS=',' read -ra DEFAULT_NODE_EXP_TARGETS <<< "$val"
|
|
||||||
fi
|
|
||||||
val=$(grep -A50 '^observability:' "$file" | grep '^ *node_exp_ips:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
|
|
||||||
if [[ -n "$val" ]]; then
|
|
||||||
IFS=',' read -ra DEFAULT_NODE_EXP_IPS <<< "$val"
|
|
||||||
fi
|
|
||||||
val=$(grep -A50 '^observability:' "$file" | grep '^ *haproxy_ips:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
|
|
||||||
if [[ -n "$val" ]]; then
|
|
||||||
IFS=',' read -ra HAPROXY_IPS <<< "$val"
|
|
||||||
fi
|
|
||||||
val=$(grep -A50 '^observability:' "$file" | grep '^ *haproxy_cluster_id:' | head -1 | sed 's/.*: *//' | tr -d '"' || true)
|
|
||||||
[[ -n "$val" ]] && HAPROXY_CLUSTER_ID="$val"
|
|
||||||
}
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Argument parsing
|
# Argument parsing
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
@ -1760,14 +1706,9 @@ parse_args() {
|
||||||
--status) ACTION="status";;
|
--status) ACTION="status";;
|
||||||
--cleanup) ACTION="cleanup";;
|
--cleanup) ACTION="cleanup";;
|
||||||
--doctor) ACTION="doctor";;
|
--doctor) ACTION="doctor";;
|
||||||
-c|--config) CONFIG_FILE="$2"; shift;;
|
-r|--remote) CLUSTER_REMOTE="$2"; shift;;
|
||||||
-r|--remote) CLUSTER_REMOTE="$2"; shift;;
|
--nodes) MANUAL_NODES="$2"; shift;;
|
||||||
--nodes) MANUAL_NODES="$2"; shift;;
|
--oc-server) OC_SERVER_IP="$2"; shift;;
|
||||||
--monitoring-target) MONITORING_TARGET="$2"; shift;;
|
|
||||||
--monitoring-ip) MONITORING_IP="$2"; shift;;
|
|
||||||
--forward-vip) FORWARD_VIP="$2"; shift;;
|
|
||||||
--oc-server) OC_SERVER_IP="$2"; shift;;
|
|
||||||
--resume) RESUME=true;;
|
|
||||||
-n|--dry-run) DRY_RUN=true;;
|
-n|--dry-run) DRY_RUN=true;;
|
||||||
-v|--verbose) VERBOSE=true;;
|
-v|--verbose) VERBOSE=true;;
|
||||||
-q|--quiet) QUIET=true;;
|
-q|--quiet) QUIET=true;;
|
||||||
|
|
@ -1788,18 +1729,6 @@ parse_args() {
|
||||||
fi
|
fi
|
||||||
}
|
}
|
||||||
|
|
||||||
require_config() {
|
|
||||||
local missing=()
|
|
||||||
[[ -z "$CLUSTER_REMOTE" ]] && missing+=("cluster_remote")
|
|
||||||
[[ -z "$FORWARD_VIP" ]] && missing+=("forward_vip")
|
|
||||||
[[ -z "$MONITORING_TARGET" ]] && missing+=("monitoring_target")
|
|
||||||
if [[ ${#missing[@]} -gt 0 ]]; then
|
|
||||||
error "Required config missing: ${missing[*]}"
|
|
||||||
error "Use --config FILE to load environment settings (see envs/)"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Main
|
# Main
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
@ -1808,16 +1737,6 @@ main() {
|
||||||
setup_colors
|
setup_colors
|
||||||
parse_args "$@"
|
parse_args "$@"
|
||||||
|
|
||||||
# Load config file if provided
|
|
||||||
if [[ -n "$CONFIG_FILE" ]]; then
|
|
||||||
parse_config "$CONFIG_FILE"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Validate required fields (must come from --config or CLI flags)
|
|
||||||
if [[ "$ACTION" != "doctor" ]]; then
|
|
||||||
require_config
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Discover cluster nodes (populates INCUS_NODES, NODE_EXP_TARGETS, NODE_EXP_IPS)
|
# Discover cluster nodes (populates INCUS_NODES, NODE_EXP_TARGETS, NODE_EXP_IPS)
|
||||||
discover_cluster_nodes
|
discover_cluster_nodes
|
||||||
|
|
||||||
|
|
@ -1,7 +1,7 @@
|
||||||
# proxmox.yaml.example - Proxmox connection config template
|
# proxmox.yaml.example - Proxmox connection config template
|
||||||
#
|
#
|
||||||
# Copy to envs/<env>/proxmox.yaml and edit with your settings.
|
# Copy to incusos/proxmox.yaml and edit with your settings.
|
||||||
# Use with: bin/incusos-proxmox --proxmox envs/<env>/proxmox.yaml
|
# This file is auto-discovered by incusos-proxmox and lab-test.
|
||||||
#
|
#
|
||||||
# Lab config files (lab-cluster.yaml, lab-oc.yaml, etc.) reference
|
# Lab config files (lab-cluster.yaml, lab-oc.yaml, etc.) reference
|
||||||
# this for connection settings, so you only need to change credentials
|
# this for connection settings, so you only need to change credentials
|
||||||
|
|
@ -78,21 +78,6 @@ EXTRA_ARGS=("${ARGS[@]:2}")
|
||||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
|
||||||
load_env() {
|
load_env() {
|
||||||
# Already exported (user sourced envs/*/env)
|
|
||||||
[[ -n "${PROXMOX_TOKEN_SECRET:-}" ]] && return
|
|
||||||
|
|
||||||
# Check envs/*/env in repo root
|
|
||||||
local repo_root
|
|
||||||
repo_root=$(cd "$SCRIPT_DIR/../.." && pwd)
|
|
||||||
for env_file in "${repo_root}"/envs/*/env; do
|
|
||||||
if [[ -f "$env_file" ]]; then
|
|
||||||
# shellcheck disable=SC1091
|
|
||||||
source "$env_file"
|
|
||||||
[[ -n "${PROXMOX_TOKEN_SECRET:-}" ]] && return
|
|
||||||
fi
|
|
||||||
done
|
|
||||||
|
|
||||||
# Walk up from script dir (backward compat)
|
|
||||||
local dir="$SCRIPT_DIR"
|
local dir="$SCRIPT_DIR"
|
||||||
while [[ "$dir" != "/" ]]; do
|
while [[ "$dir" != "/" ]]; do
|
||||||
if [[ -f "${dir}/env" ]]; then
|
if [[ -f "${dir}/env" ]]; then
|
||||||
|
|
@ -124,21 +109,9 @@ yaml_get() {
|
||||||
}
|
}
|
||||||
|
|
||||||
find_proxmox_yaml() {
|
find_proxmox_yaml() {
|
||||||
local repo_root
|
|
||||||
repo_root=$(cd "$SCRIPT_DIR/../.." && pwd)
|
|
||||||
|
|
||||||
# Check envs/*/proxmox.yaml in repo root
|
|
||||||
for f in "${repo_root}"/envs/*/proxmox.yaml; do
|
|
||||||
if [[ -f "$f" ]]; then
|
|
||||||
echo "$f"
|
|
||||||
return
|
|
||||||
fi
|
|
||||||
done
|
|
||||||
|
|
||||||
# Walk up from script dir (backward compat)
|
|
||||||
local search_dirs=(
|
local search_dirs=(
|
||||||
"${SCRIPT_DIR}/.."
|
"${SCRIPT_DIR}/.."
|
||||||
"${repo_root}"
|
"${SCRIPT_DIR}/../.."
|
||||||
"$(pwd)"
|
"$(pwd)"
|
||||||
)
|
)
|
||||||
for dir in "${search_dirs[@]}"; do
|
for dir in "${search_dirs[@]}"; do
|
||||||
|
|
@ -146,6 +119,10 @@ find_proxmox_yaml() {
|
||||||
echo "${dir}/proxmox.yaml"
|
echo "${dir}/proxmox.yaml"
|
||||||
return
|
return
|
||||||
fi
|
fi
|
||||||
|
if [[ -f "${dir}/incusos/proxmox.yaml" ]]; then
|
||||||
|
echo "${dir}/incusos/proxmox.yaml"
|
||||||
|
return
|
||||||
|
fi
|
||||||
done
|
done
|
||||||
echo ""
|
echo ""
|
||||||
}
|
}
|
||||||
|
|
@ -71,21 +71,6 @@ fi
|
||||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
|
||||||
load_env() {
|
load_env() {
|
||||||
# Already exported (user sourced envs/*/env)
|
|
||||||
[[ -n "${PROXMOX_ROOT_PASSWORD:-}" ]] && return
|
|
||||||
|
|
||||||
# Check envs/*/env in repo root
|
|
||||||
local repo_root
|
|
||||||
repo_root=$(cd "$SCRIPT_DIR/../.." && pwd)
|
|
||||||
for env_file in "${repo_root}"/envs/*/env; do
|
|
||||||
if [[ -f "$env_file" ]]; then
|
|
||||||
# shellcheck disable=SC1091
|
|
||||||
source "$env_file"
|
|
||||||
[[ -n "${PROXMOX_ROOT_PASSWORD:-}" ]] && return
|
|
||||||
fi
|
|
||||||
done
|
|
||||||
|
|
||||||
# Walk up from script dir (backward compat)
|
|
||||||
local dir="$SCRIPT_DIR"
|
local dir="$SCRIPT_DIR"
|
||||||
while [[ "$dir" != "/" ]]; do
|
while [[ "$dir" != "/" ]]; do
|
||||||
if [[ -f "${dir}/env" ]]; then
|
if [[ -f "${dir}/env" ]]; then
|
||||||
|
|
@ -117,21 +102,10 @@ yaml_get() {
|
||||||
}
|
}
|
||||||
|
|
||||||
find_proxmox_yaml() {
|
find_proxmox_yaml() {
|
||||||
local repo_root
|
# Search: script's grandparent (repo/incusos -> repo), script parent, cwd
|
||||||
repo_root=$(cd "$SCRIPT_DIR/../.." && pwd)
|
|
||||||
|
|
||||||
# Check envs/*/proxmox.yaml in repo root
|
|
||||||
for f in "${repo_root}"/envs/*/proxmox.yaml; do
|
|
||||||
if [[ -f "$f" ]]; then
|
|
||||||
echo "$f"
|
|
||||||
return
|
|
||||||
fi
|
|
||||||
done
|
|
||||||
|
|
||||||
# Walk up from script dir (backward compat)
|
|
||||||
local search_dirs=(
|
local search_dirs=(
|
||||||
"${SCRIPT_DIR}/.."
|
"${SCRIPT_DIR}/.."
|
||||||
"${repo_root}"
|
"${SCRIPT_DIR}/../.."
|
||||||
"$(pwd)"
|
"$(pwd)"
|
||||||
)
|
)
|
||||||
for dir in "${search_dirs[@]}"; do
|
for dir in "${search_dirs[@]}"; do
|
||||||
|
|
@ -139,6 +113,10 @@ find_proxmox_yaml() {
|
||||||
echo "${dir}/proxmox.yaml"
|
echo "${dir}/proxmox.yaml"
|
||||||
return
|
return
|
||||||
fi
|
fi
|
||||||
|
if [[ -f "${dir}/incusos/proxmox.yaml" ]]; then
|
||||||
|
echo "${dir}/incusos/proxmox.yaml"
|
||||||
|
return
|
||||||
|
fi
|
||||||
done
|
done
|
||||||
echo ""
|
echo ""
|
||||||
}
|
}
|
||||||
|
|
@ -22,10 +22,10 @@ readonly MANAGED_MARKER="[incusos-lab:managed]"
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Auto-load env file (secrets like PROXMOX_TOKEN_SECRET)
|
# Auto-load env file (secrets like PROXMOX_TOKEN_SECRET)
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Searches for an 'env' file. Sources it if found and not already loaded.
|
# Searches for an 'env' file: first next to the script, then walking up from
|
||||||
# Priority: already exported > envs/*/env in repo > walk up from cwd.
|
# the current directory. Sources it if found and not already loaded. This is
|
||||||
# This is a convenience -- users can still export variables manually or
|
# a convenience -- users can still export variables manually or source the
|
||||||
# source the file themselves (e.g., source envs/hetzner/env).
|
# file themselves.
|
||||||
|
|
||||||
load_env_file() {
|
load_env_file() {
|
||||||
# Skip if the secret is already set (user exported it, or env already sourced)
|
# Skip if the secret is already set (user exported it, or env already sourced)
|
||||||
|
|
@ -35,24 +35,18 @@ load_env_file() {
|
||||||
|
|
||||||
local script_real
|
local script_real
|
||||||
script_real=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
script_real=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
||||||
local repo_root
|
|
||||||
repo_root=$(cd "$script_real/.." && pwd)
|
|
||||||
|
|
||||||
# Check: envs/*/env in repo root (per-environment secrets)
|
# Check: script directory's parent (repo root when script is in incusos/)
|
||||||
for env_file in "${repo_root}"/envs/*/env; do
|
if [[ -f "${script_real}/../env" ]]; then
|
||||||
if [[ -f "$env_file" ]]; then
|
|
||||||
# shellcheck source=/dev/null
|
|
||||||
source "$env_file"
|
|
||||||
if [[ -n "${PROXMOX_TOKEN_SECRET:-}" ]]; then
|
|
||||||
return 0
|
|
||||||
fi
|
|
||||||
fi
|
|
||||||
done
|
|
||||||
|
|
||||||
# Check: repo root env file (backward compat)
|
|
||||||
if [[ -f "${repo_root}/env" ]]; then
|
|
||||||
# shellcheck source=/dev/null
|
# shellcheck source=/dev/null
|
||||||
source "${repo_root}/env"
|
source "${script_real}/../env"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Check: script directory itself
|
||||||
|
if [[ -f "${script_real}/env" ]]; then
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source "${script_real}/env"
|
||||||
return 0
|
return 0
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
|
@ -1411,13 +1405,6 @@ json.dump(result, sys.stdout, indent=2)
|
||||||
val=$(json_get "$px_json" ".api_token_id" ""); [[ -n "$val" ]] && PVE_API_TOKEN_ID="$val"
|
val=$(json_get "$px_json" ".api_token_id" ""); [[ -n "$val" ]] && PVE_API_TOKEN_ID="$val"
|
||||||
val=$(json_get "$px_json" ".pool" ""); [[ -n "$val" ]] && PVE_POOL="$val"
|
val=$(json_get "$px_json" ".pool" ""); [[ -n "$val" ]] && PVE_POOL="$val"
|
||||||
val=$(json_get "$px_json" ".method" ""); [[ -n "$val" ]] && PVE_METHOD="$val"
|
val=$(json_get "$px_json" ".method" ""); [[ -n "$val" ]] && PVE_METHOD="$val"
|
||||||
# api_token_secret in proxmox.yaml overrides the repo-wide env file, useful
|
|
||||||
# for per-target configs where the env file holds a different target's token.
|
|
||||||
val=$(json_get "$px_json" ".api_token_secret" "")
|
|
||||||
if [[ -n "$val" ]]; then
|
|
||||||
export PROXMOX_TOKEN_SECRET="$val"
|
|
||||||
detail "Using api_token_secret from proxmox.yaml"
|
|
||||||
fi
|
|
||||||
|
|
||||||
rm -f "$px_json"
|
rm -f "$px_json"
|
||||||
}
|
}
|
||||||
|
|
@ -2294,7 +2281,6 @@ build_vm_description() {
|
||||||
# Check if a VM has our management marker in its description
|
# Check if a VM has our management marker in its description
|
||||||
pve_vm_is_managed() {
|
pve_vm_is_managed() {
|
||||||
local vmid="$1"
|
local vmid="$1"
|
||||||
local encoded_marker="${MANAGED_MARKER//:/%3A}"
|
|
||||||
|
|
||||||
if [[ "$DRY_RUN" == true ]]; then
|
if [[ "$DRY_RUN" == true ]]; then
|
||||||
return 1
|
return 1
|
||||||
|
|
@ -2310,8 +2296,7 @@ pve_vm_is_managed() {
|
||||||
;;
|
;;
|
||||||
esac
|
esac
|
||||||
|
|
||||||
echo "$config_output" | grep -qF "$MANAGED_MARKER" || \
|
echo "$config_output" | grep -qF "$MANAGED_MARKER"
|
||||||
echo "$config_output" | grep -qF "$encoded_marker"
|
|
||||||
}
|
}
|
||||||
|
|
||||||
# Check if a VM still has install media (ide2) attached.
|
# Check if a VM still has install media (ide2) attached.
|
||||||
|
|
@ -2412,15 +2397,15 @@ preflight_checks() {
|
||||||
if [[ "$DRY_RUN" != true ]]; then
|
if [[ "$DRY_RUN" != true ]]; then
|
||||||
case "$PVE_METHOD" in
|
case "$PVE_METHOD" in
|
||||||
ssh)
|
ssh)
|
||||||
if ! pve_run "pvesm status 2>/dev/null" | awk 'NR>1 {print $1}' | grep -qw "$PVE_STORAGE"; then
|
if ! pve_run "pvesm status 2>/dev/null" | awk 'NR>1 {print \$1}' | grep -qw "$PVE_STORAGE"; then
|
||||||
error "Storage pool '${PVE_STORAGE}' not found on Proxmox"
|
error "Storage pool '${PVE_STORAGE}' not found on Proxmox"
|
||||||
error "Available pools: $(pve_run "pvesm status 2>/dev/null" | awk 'NR>1 {print $1}' | tr '\n' ' ')"
|
error "Available pools: $(pve_run "pvesm status 2>/dev/null" | awk 'NR>1 {print \$1}' | tr '\n' ' ')"
|
||||||
error "Fix: update 'proxmox.storage' in your config file"
|
error "Fix: update 'proxmox.storage' in your config file"
|
||||||
errors=$((errors + 1))
|
errors=$((errors + 1))
|
||||||
fi
|
fi
|
||||||
if ! pve_run "pvesm status 2>/dev/null" | awk 'NR>1 {print $1}' | grep -qw "$PVE_ISO_STORAGE"; then
|
if ! pve_run "pvesm status 2>/dev/null" | awk 'NR>1 {print \$1}' | grep -qw "$PVE_ISO_STORAGE"; then
|
||||||
error "ISO storage '${PVE_ISO_STORAGE}' not found on Proxmox"
|
error "ISO storage '${PVE_ISO_STORAGE}' not found on Proxmox"
|
||||||
error "Available pools: $(pve_run "pvesm status 2>/dev/null" | awk 'NR>1 {print $1}' | tr '\n' ' ')"
|
error "Available pools: $(pve_run "pvesm status 2>/dev/null" | awk 'NR>1 {print \$1}' | tr '\n' ' ')"
|
||||||
error "Fix: update 'proxmox.iso_storage' in your config file"
|
error "Fix: update 'proxmox.iso_storage' in your config file"
|
||||||
errors=$((errors + 1))
|
errors=$((errors + 1))
|
||||||
fi
|
fi
|
||||||
|
|
@ -163,20 +163,6 @@ done
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
load_env() {
|
load_env() {
|
||||||
[[ -n "${PROXMOX_TOKEN_SECRET:-}" ]] && return
|
|
||||||
|
|
||||||
# Check envs/*/env in repo root
|
|
||||||
local repo_root
|
|
||||||
repo_root=$(cd "$SCRIPT_DIR/../.." && pwd)
|
|
||||||
for env_file in "${repo_root}"/envs/*/env; do
|
|
||||||
if [[ -f "$env_file" ]]; then
|
|
||||||
# shellcheck disable=SC1091
|
|
||||||
source "$env_file"
|
|
||||||
[[ -n "${PROXMOX_TOKEN_SECRET:-}" ]] && return
|
|
||||||
fi
|
|
||||||
done
|
|
||||||
|
|
||||||
# Walk up from script dir (backward compat)
|
|
||||||
local dir="$SCRIPT_DIR"
|
local dir="$SCRIPT_DIR"
|
||||||
while [[ "$dir" != "/" ]]; do
|
while [[ "$dir" != "/" ]]; do
|
||||||
if [[ -f "${dir}/env" ]]; then
|
if [[ -f "${dir}/env" ]]; then
|
||||||
|
|
@ -86,16 +86,6 @@ done
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
load_env() {
|
load_env() {
|
||||||
[[ -n "${PROXMOX_TOKEN_SECRET:-}" ]] && return
|
|
||||||
|
|
||||||
# Check envs/*/env in repo root
|
|
||||||
local repo_root
|
|
||||||
repo_root=$(cd "$SCRIPT_DIR/../.." && pwd)
|
|
||||||
for env_file in "${repo_root}"/envs/*/env; do
|
|
||||||
[[ -f "$env_file" ]] && { source "$env_file"; [[ -n "${PROXMOX_TOKEN_SECRET:-}" ]] && return; }
|
|
||||||
done
|
|
||||||
|
|
||||||
# Walk up from script dir (backward compat)
|
|
||||||
local dir="$SCRIPT_DIR"
|
local dir="$SCRIPT_DIR"
|
||||||
while [[ "$dir" != "/" ]]; do
|
while [[ "$dir" != "/" ]]; do
|
||||||
[[ -f "${dir}/env" ]] && { source "${dir}/env"; return; }
|
[[ -f "${dir}/env" ]] && { source "${dir}/env"; return; }
|
||||||
|
|
@ -107,21 +97,9 @@ load_env
|
||||||
[[ -z "${PROXMOX_TOKEN_SECRET:-}" ]] && { error "PROXMOX_TOKEN_SECRET not set"; exit 1; }
|
[[ -z "${PROXMOX_TOKEN_SECRET:-}" ]] && { error "PROXMOX_TOKEN_SECRET not set"; exit 1; }
|
||||||
[[ -z "${PROXMOX_ROOT_PASSWORD:-}" ]] && { error "PROXMOX_ROOT_PASSWORD not set"; exit 1; }
|
[[ -z "${PROXMOX_ROOT_PASSWORD:-}" ]] && { error "PROXMOX_ROOT_PASSWORD not set"; exit 1; }
|
||||||
|
|
||||||
find_proxmox_yaml() {
|
|
||||||
local repo_root
|
|
||||||
repo_root=$(cd "$SCRIPT_DIR/../.." && pwd)
|
|
||||||
for f in "${repo_root}"/envs/*/proxmox.yaml; do
|
|
||||||
[[ -f "$f" ]] && { echo "$f"; return; }
|
|
||||||
done
|
|
||||||
[[ -f "${repo_root}/proxmox.yaml" ]] && { echo "${repo_root}/proxmox.yaml"; return; }
|
|
||||||
echo ""
|
|
||||||
}
|
|
||||||
PROXMOX_YAML=$(find_proxmox_yaml)
|
|
||||||
[[ -z "$PROXMOX_YAML" ]] && { error "proxmox.yaml not found"; exit 1; }
|
|
||||||
|
|
||||||
yaml_get() {
|
yaml_get() {
|
||||||
local val
|
local val
|
||||||
val=$(awk -F': ' -v k="$1" '$1 == k {print $2; exit}' "$PROXMOX_YAML" 2>/dev/null) || val=""
|
val=$(awk -F': ' -v k="$1" '$1 == k {print $2; exit}' "${SCRIPT_DIR}/proxmox.yaml" 2>/dev/null) || val=""
|
||||||
val="${val#\"}" ; val="${val%\"}" ; val="${val#\'}" ; val="${val%\'}"
|
val="${val#\"}" ; val="${val%\"}" ; val="${val#\'}" ; val="${val%\'}"
|
||||||
echo "${val:-${2:-}}"
|
echo "${val:-${2:-}}"
|
||||||
}
|
}
|
||||||
|
|
@ -470,7 +448,7 @@ for ((i=0; i<COUNT; i++)); do
|
||||||
import subprocess, os, sys
|
import subprocess, os, sys
|
||||||
# Read token_id from proxmox.yaml
|
# Read token_id from proxmox.yaml
|
||||||
token_id = ''
|
token_id = ''
|
||||||
yaml_file = '${PROXMOX_YAML}'
|
yaml_file = '${SCRIPT_DIR}/proxmox.yaml'
|
||||||
for line in open(yaml_file):
|
for line in open(yaml_file):
|
||||||
line = line.strip()
|
line = line.strip()
|
||||||
if line.startswith('api_token_id:'):
|
if line.startswith('api_token_id:'):
|
||||||
|
|
@ -13,7 +13,7 @@ set -euo pipefail
|
||||||
readonly VERSION="0.1.0"
|
readonly VERSION="0.1.0"
|
||||||
readonly SCRIPT_NAME="manage-dashboards"
|
readonly SCRIPT_NAME="manage-dashboards"
|
||||||
readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
readonly DASHBOARDS_DIR="${SCRIPT_DIR}/data/observability-dashboards"
|
readonly DASHBOARDS_DIR="${SCRIPT_DIR}/observability-dashboards"
|
||||||
readonly MANIFEST_FILE="${DASHBOARDS_DIR}/dashboard.yaml"
|
readonly MANIFEST_FILE="${DASHBOARDS_DIR}/dashboard.yaml"
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
@ -30,8 +30,7 @@ set -euo pipefail
|
||||||
readonly VERSION="0.1.0"
|
readonly VERSION="0.1.0"
|
||||||
readonly SCRIPT_NAME="observe-deploy"
|
readonly SCRIPT_NAME="observe-deploy"
|
||||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
BIN_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
|
SEED_TOOL="${SCRIPT_DIR}/incusos-seed"
|
||||||
SEED_TOOL="${BIN_DIR}/incusos-seed"
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Defaults
|
# Defaults
|
||||||
|
|
@ -185,20 +184,6 @@ done
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
load_env() {
|
load_env() {
|
||||||
[[ -n "${PROXMOX_TOKEN_SECRET:-}" ]] && return
|
|
||||||
|
|
||||||
# Check envs/*/env in repo root
|
|
||||||
local repo_root
|
|
||||||
repo_root=$(cd "$SCRIPT_DIR/../.." && pwd)
|
|
||||||
for env_file in "${repo_root}"/envs/*/env; do
|
|
||||||
if [[ -f "$env_file" ]]; then
|
|
||||||
# shellcheck disable=SC1091
|
|
||||||
source "$env_file"
|
|
||||||
[[ -n "${PROXMOX_TOKEN_SECRET:-}" ]] && return
|
|
||||||
fi
|
|
||||||
done
|
|
||||||
|
|
||||||
# Walk up from script dir (backward compat)
|
|
||||||
local dir="$SCRIPT_DIR"
|
local dir="$SCRIPT_DIR"
|
||||||
while [[ "$dir" != "/" ]]; do
|
while [[ "$dir" != "/" ]]; do
|
||||||
if [[ -f "${dir}/env" ]]; then
|
if [[ -f "${dir}/env" ]]; then
|
||||||
|
|
@ -237,27 +222,13 @@ yaml_get() {
|
||||||
|
|
||||||
load_proxmox_yaml() {
|
load_proxmox_yaml() {
|
||||||
local yaml_file=""
|
local yaml_file=""
|
||||||
local repo_root
|
if [[ -f "${SCRIPT_DIR}/proxmox.yaml" ]]; then
|
||||||
repo_root=$(cd "$SCRIPT_DIR/../.." && pwd)
|
yaml_file="${SCRIPT_DIR}/proxmox.yaml"
|
||||||
|
elif [[ -f "proxmox.yaml" ]]; then
|
||||||
# Check envs/*/proxmox.yaml
|
yaml_file="proxmox.yaml"
|
||||||
for f in "${repo_root}"/envs/*/proxmox.yaml; do
|
else
|
||||||
if [[ -f "$f" ]]; then
|
error "proxmox.yaml not found"
|
||||||
yaml_file="$f"
|
exit 1
|
||||||
break
|
|
||||||
fi
|
|
||||||
done
|
|
||||||
|
|
||||||
# Backward compat: repo root or cwd
|
|
||||||
if [[ -z "$yaml_file" ]]; then
|
|
||||||
if [[ -f "${repo_root}/proxmox.yaml" ]]; then
|
|
||||||
yaml_file="${repo_root}/proxmox.yaml"
|
|
||||||
elif [[ -f "proxmox.yaml" ]]; then
|
|
||||||
yaml_file="proxmox.yaml"
|
|
||||||
else
|
|
||||||
error "proxmox.yaml not found"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
fi
|
fi
|
||||||
|
|
||||||
PVE_HOST=$(yaml_get "$yaml_file" "host")
|
PVE_HOST=$(yaml_get "$yaml_file" "host")
|
||||||
|
|
@ -1055,7 +1026,7 @@ setup_output() {
|
||||||
fi
|
fi
|
||||||
fi
|
fi
|
||||||
|
|
||||||
RUN_DIR="${BIN_DIR}/diagnostic/observe-runs/${timestamp}_${LABEL}"
|
RUN_DIR="${SCRIPT_DIR}/observe-runs/${timestamp}_${LABEL}"
|
||||||
mkdir -p "$RUN_DIR"
|
mkdir -p "$RUN_DIR"
|
||||||
|
|
||||||
LOG_FILE="${RUN_DIR}/deploy.log"
|
LOG_FILE="${RUN_DIR}/deploy.log"
|
||||||
|
|
@ -1344,7 +1315,7 @@ else
|
||||||
echo ""
|
echo ""
|
||||||
|
|
||||||
# Output directories
|
# Output directories
|
||||||
echo " Output: ${BIN_DIR}/diagnostic/observe-runs/*${BASE_LABEL}*"
|
echo " Output: ${SCRIPT_DIR}/observe-runs/*${BASE_LABEL}*"
|
||||||
echo ""
|
echo ""
|
||||||
|
|
||||||
if [[ $total_fail -gt 0 ]]; then
|
if [[ $total_fail -gt 0 ]]; then
|
||||||
|
|
@ -0,0 +1,86 @@
|
||||||
|
# Target Configurations
|
||||||
|
|
||||||
|
Target-specific Proxmox connection configs and lab definitions. Each
|
||||||
|
subdirectory represents a physical host (or host type) with its own
|
||||||
|
network topology, storage backend, and resource sizing.
|
||||||
|
|
||||||
|
## How it works
|
||||||
|
|
||||||
|
`incusos-proxmox` already supports `--proxmox FILE` to point at any
|
||||||
|
connection config. Targets are just an organizational convention --
|
||||||
|
no script changes required.
|
||||||
|
|
||||||
|
## Directory layout
|
||||||
|
|
||||||
|
```
|
||||||
|
targets/
|
||||||
|
├── beelink/
|
||||||
|
│ ├── proxmox.yaml.example # Connection template (LAN, vmbr0, local-lvm)
|
||||||
|
│ └── lab-cluster.yaml # 3-node cluster sized for beelink (4C/4G/50G)
|
||||||
|
└── hetzner/
|
||||||
|
├── proxmox.yaml.example # Connection template (WireGuard, vmbr1, local-zfs)
|
||||||
|
├── lab-cluster.yaml # 3-node cluster sized for hetzner (8C/16G/100G)
|
||||||
|
└── lab-production.yaml # OC + 3-node cluster for hetzner
|
||||||
|
```
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
### 1. Copy the example and fill in credentials
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cp incusos/targets/hetzner/proxmox.yaml.example incusos/targets/hetzner/proxmox.yaml
|
||||||
|
# Edit with your API token, host IP, etc.
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Deploy with explicit --proxmox
|
||||||
|
|
||||||
|
```bash
|
||||||
|
incusos-proxmox --proxmox incusos/targets/hetzner/proxmox.yaml \
|
||||||
|
incusos/targets/hetzner/lab-cluster.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Or symlink for convenience
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Point the auto-discovered proxmox.yaml at your active target
|
||||||
|
ln -sf targets/hetzner/proxmox.yaml incusos/proxmox.yaml
|
||||||
|
|
||||||
|
# Now deploy without --proxmox
|
||||||
|
incusos-proxmox incusos/targets/hetzner/lab-cluster.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
## Environment variables
|
||||||
|
|
||||||
|
`incusos-proxmox` reads `PROXMOX_TOKEN_SECRET` from the environment (or
|
||||||
|
the `env` file at the repo root). When working with multiple targets,
|
||||||
|
use different variable names per host:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# In env file
|
||||||
|
PROXMOX_TOKEN_SECRET=beelink-token-here
|
||||||
|
HETZNER_PROXMOX_TOKEN_SECRET=hetzner-token-here
|
||||||
|
```
|
||||||
|
|
||||||
|
Before deploying to a specific target, export the right secret:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Deploy to Hetzner
|
||||||
|
export PROXMOX_TOKEN_SECRET="$HETZNER_PROXMOX_TOKEN_SECRET"
|
||||||
|
incusos-proxmox --proxmox incusos/targets/hetzner/proxmox.yaml \
|
||||||
|
incusos/targets/hetzner/lab-cluster.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
## Key differences between targets
|
||||||
|
|
||||||
|
| Setting | Beelink | Hetzner |
|
||||||
|
|---------|---------|---------|
|
||||||
|
| Host | 192.168.1.10 (LAN) | 10.10.0.1 (WireGuard) |
|
||||||
|
| Method | ssh | api |
|
||||||
|
| Bridge | vmbr0 | vmbr1 (private) |
|
||||||
|
| Storage | local-lvm | local-zfs |
|
||||||
|
| VLAN | 69 | *(none)* |
|
||||||
|
| Gateway | 192.168.100.1 | 10.10.0.1 |
|
||||||
|
| VM IPs | 192.168.102.x/22 | 10.10.0.x/24 |
|
||||||
|
| VM cores | 4 | 8 |
|
||||||
|
| VM memory | 4-8 GiB | 16 GiB |
|
||||||
|
| VM disk | 50 GiB | 100 GiB |
|
||||||
|
|
@ -4,8 +4,8 @@
|
||||||
# Deploys 3 standalone Incus nodes that can be clustered after deployment.
|
# Deploys 3 standalone Incus nodes that can be clustered after deployment.
|
||||||
#
|
#
|
||||||
# Usage:
|
# Usage:
|
||||||
# bin/incusos-proxmox --proxmox envs/beelink/proxmox.yaml \
|
# incusos-proxmox --proxmox incusos/targets/beelink/proxmox.yaml \
|
||||||
# envs/beelink/lab-cluster.yaml
|
# incusos/targets/beelink/lab-cluster.yaml
|
||||||
#
|
#
|
||||||
# Connection settings from proxmox.yaml in this directory.
|
# Connection settings from proxmox.yaml in this directory.
|
||||||
|
|
||||||
|
|
@ -7,8 +7,8 @@
|
||||||
# VMs get static IPs in the 10.10.0.101-103 range.
|
# VMs get static IPs in the 10.10.0.101-103 range.
|
||||||
#
|
#
|
||||||
# Usage:
|
# Usage:
|
||||||
# bin/incusos-proxmox --proxmox envs/hetzner/proxmox.yaml \
|
# incusos-proxmox --proxmox incusos/targets/hetzner/proxmox.yaml \
|
||||||
# envs/hetzner/lab-cluster.yaml
|
# incusos/targets/hetzner/lab-cluster.yaml
|
||||||
#
|
#
|
||||||
# Connection settings from proxmox.yaml in this directory.
|
# Connection settings from proxmox.yaml in this directory.
|
||||||
|
|
||||||
|
|
@ -1,6 +1,6 @@
|
||||||
# lab-production.yaml - Production-quality lab for Hetzner (OC + 3-node cluster)
|
# lab-production.yaml - Production-quality lab for Hetzner (OC + 3-node cluster)
|
||||||
#
|
#
|
||||||
# Deploys 1 Operations Center instance and 3 Incus nodes for OC-managed clustering.
|
# Deploys 1 Operations Center instance and 3 Incus nodes for manual clustering.
|
||||||
# Sized for a Hetzner dedicated server with generous resources.
|
# Sized for a Hetzner dedicated server with generous resources.
|
||||||
#
|
#
|
||||||
# Architecture:
|
# Architecture:
|
||||||
|
|
@ -12,11 +12,11 @@
|
||||||
# Network: 10.10.0.0/24 on vmbr1 (private bridge, NAT to internet).
|
# Network: 10.10.0.0/24 on vmbr1 (private bridge, NAT to internet).
|
||||||
# OVN overlay (once configured): 10.10.10.0/24 internal, 10.10.0.200+ external.
|
# OVN overlay (once configured): 10.10.10.0/24 internal, 10.10.0.200+ external.
|
||||||
#
|
#
|
||||||
# RAM budget: ~248 GiB of 256 GiB (8 GiB OC + 3×80 GiB nodes; ~8 GiB reserved for Proxmox host)
|
# RAM budget: 56 GiB of 256 GiB (22% utilization -- plenty of room for workloads)
|
||||||
#
|
#
|
||||||
# Usage:
|
# Usage:
|
||||||
# bin/incusos-proxmox --proxmox envs/hetzner/proxmox.yaml \
|
# incusos-proxmox --proxmox incusos/targets/hetzner/proxmox.yaml \
|
||||||
# envs/hetzner/lab-production.yaml
|
# incusos/targets/hetzner/lab-production.yaml
|
||||||
#
|
#
|
||||||
# After deployment, follow notes/production-lab-guide.md for:
|
# After deployment, follow notes/production-lab-guide.md for:
|
||||||
# - Cluster formation (section 4)
|
# - Cluster formation (section 4)
|
||||||
|
|
@ -26,8 +26,8 @@
|
||||||
# Connection settings from proxmox.yaml in this directory.
|
# Connection settings from proxmox.yaml in this directory.
|
||||||
|
|
||||||
defaults:
|
defaults:
|
||||||
cores: 16
|
cores: 8
|
||||||
memory: 81920 # ~80 GiB per node (3 nodes × 80 = 240 GiB + 8 for OC = 248 GiB total)
|
memory: 16384
|
||||||
disk: 100
|
disk: 100
|
||||||
start_vmid: 910
|
start_vmid: 910
|
||||||
|
|
||||||
|
|
@ -36,12 +36,12 @@ vms:
|
||||||
app: operations-center
|
app: operations-center
|
||||||
apply_defaults: true
|
apply_defaults: true
|
||||||
cores: 4
|
cores: 4
|
||||||
memory: 8192 # OC needs very little RAM
|
memory: 8192
|
||||||
ip: 10.10.0.110/24
|
ip: 10.10.0.110/24
|
||||||
|
|
||||||
- name: hz-node-01
|
- name: hz-node-01
|
||||||
app: incus
|
app: incus
|
||||||
apply_defaults: false # OC manages cluster formation + storage/network setup
|
apply_defaults: true # init node: needs storage pool + network
|
||||||
disk: 150 # extra space for OVN control plane container
|
disk: 150 # extra space for OVN control plane container
|
||||||
ip: 10.10.0.111/24
|
ip: 10.10.0.111/24
|
||||||
|
|
||||||
|
|
@ -9,12 +9,11 @@
|
||||||
#
|
#
|
||||||
# Storage: local-zfs (ZFS pool on dedicated disks).
|
# Storage: local-zfs (ZFS pool on dedicated disks).
|
||||||
#
|
#
|
||||||
# See envs/hetzner/setup/hetzner-setup.md for the full setup guide.
|
# See hetzner/hetzner-setup.md for the full setup guide.
|
||||||
|
|
||||||
host: 10.10.0.1 # Proxmox host IP (via WireGuard, on vmbr1)
|
host: 10.10.0.1 # Proxmox host IP (via WireGuard, on vmbr1)
|
||||||
method: api # api recommended (WireGuard may add SSH latency)
|
method: api # api recommended (WireGuard may add SSH latency)
|
||||||
api_token_id: automation@pve!deploy
|
api_token_id: automation@pve!deploy
|
||||||
api_token_secret: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx # token secret (overrides env PROXMOX_TOKEN_SECRET)
|
|
||||||
node: pve # Proxmox node name (adjust if different)
|
node: pve # Proxmox node name (adjust if different)
|
||||||
storage: local-zfs # VM disk storage (ZFS pool)
|
storage: local-zfs # VM disk storage (ZFS pool)
|
||||||
iso_storage: local # ISO/image storage
|
iso_storage: local # ISO/image storage
|
||||||
|
|
@ -97,21 +97,8 @@ done
|
||||||
# Load environment
|
# Load environment
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
# Auto-load env file for PROXMOX_TOKEN_SECRET
|
||||||
load_env() {
|
load_env() {
|
||||||
[[ -n "${PROXMOX_TOKEN_SECRET:-}" ]] && return
|
|
||||||
|
|
||||||
# Check envs/*/env in repo root
|
|
||||||
local repo_root
|
|
||||||
repo_root=$(cd "$SCRIPT_DIR/../.." && pwd)
|
|
||||||
for env_file in "${repo_root}"/envs/*/env; do
|
|
||||||
if [[ -f "$env_file" ]]; then
|
|
||||||
# shellcheck disable=SC1091
|
|
||||||
source "$env_file"
|
|
||||||
[[ -n "${PROXMOX_TOKEN_SECRET:-}" ]] && return
|
|
||||||
fi
|
|
||||||
done
|
|
||||||
|
|
||||||
# Walk up from script dir (backward compat)
|
|
||||||
local dir="$SCRIPT_DIR"
|
local dir="$SCRIPT_DIR"
|
||||||
while [[ "$dir" != "/" ]]; do
|
while [[ "$dir" != "/" ]]; do
|
||||||
if [[ -f "${dir}/env" ]]; then
|
if [[ -f "${dir}/env" ]]; then
|
||||||
|
|
@ -1,131 +0,0 @@
|
||||||
# Aether Inventory After Redeploy (Without DB Backup)
|
|
||||||
|
|
||||||
**Date**: 2026-03-04
|
|
||||||
**Target**: Hetzner lab, Aether at 10.10.0.120
|
|
||||||
**Version**: 6.4.440 (up from 6.4.202 at time of redeploy)
|
|
||||||
**Activation code**: 443F4EEC6E088017 (new — DB reinstall generates a new one)
|
|
||||||
|
|
||||||
## Summary
|
|
||||||
|
|
||||||
Aether was redeployed from a golden image without restoring the previous
|
|
||||||
database. This inventory documents what survived (auto-discovered from
|
|
||||||
the live cluster) versus what was lost (stored only in the DB).
|
|
||||||
|
|
||||||
### What survived (auto-discovered from cluster)
|
|
||||||
|
|
||||||
These are things Aether reconstructs by connecting to the Incus cluster
|
|
||||||
and querying the API. They require no database state.
|
|
||||||
|
|
||||||
| Feature | Status | Detail |
|
|
||||||
|---------|--------|--------|
|
|
||||||
| Cluster connection | **Present** | hz-cluster (ID 53, was 52 before redeploy) |
|
|
||||||
| Cluster members | **Present** | 3/3 nodes online, fully operational |
|
|
||||||
| Instance inventory | **Present** | 12 instances, all Running, correct IPs and locations |
|
|
||||||
| Instance tags | **Present** | HAProxy tags (ha-group, managed, role) visible |
|
|
||||||
| Storage pools | **Present** | local (zfs), 6% used (7GB/112.4GB) |
|
|
||||||
| Networking/OVN | **Present** | Visible via Manage INCUS Clusters > Networking |
|
|
||||||
| Cluster health | **Present** | Health dashboard shows CPU/mem/load per node |
|
|
||||||
|
|
||||||
**Key insight**: Aether's core value — seeing your cluster and instances —
|
|
||||||
works immediately after a fresh deploy + cluster add. The Incus API is the
|
|
||||||
source of truth for all infrastructure state; Aether caches it but can
|
|
||||||
rebuild from scratch.
|
|
||||||
|
|
||||||
### What was lost (DB-only state)
|
|
||||||
|
|
||||||
These are things Aether stores in its PostgreSQL database that have no
|
|
||||||
external source of truth. They must be manually recreated.
|
|
||||||
|
|
||||||
| Feature | Status | Detail |
|
|
||||||
|---------|--------|--------|
|
|
||||||
| Blueprints | **Empty** | "No blueprints created yet" — all designs gone |
|
|
||||||
| Deployed blueprints | **Empty** | No deployment history — tracking of which blueprints were deployed where is lost |
|
|
||||||
| Operations Center connections | **Empty** | No OC configured — previous OC link gone |
|
|
||||||
| Global firewall rules | **Empty** | 0 rules (API confirms: `[]`) |
|
|
||||||
| Cluster firewall rules | **Empty** | 0 rules (API confirms: `[]`) |
|
|
||||||
| RBAC users/roles | **Default only** | Only the built-in admin account exists |
|
|
||||||
| License | **Gone** | New activation code generated — previous license key invalid |
|
|
||||||
| Ledger / audit log | **Empty** | No historical log entries |
|
|
||||||
| Sync logs | **Empty** | No sync history |
|
|
||||||
| AWX job history | **Empty** | "No AWX job history yet" — past job runs not tracked |
|
|
||||||
| HAProxy image push state | **Not pushed** | Image v2.64 exists locally but shows "No Image" for hz-cluster — needs re-push |
|
|
||||||
| Auto-backup setting | **Off** | `auto_backup_enabled: False` (default) |
|
|
||||||
|
|
||||||
### What was re-added manually (post-redeploy)
|
|
||||||
|
|
||||||
These items exist in the current database because they were manually
|
|
||||||
reconfigured after the redeploy.
|
|
||||||
|
|
||||||
| Feature | Status | Detail |
|
|
||||||
|---------|--------|--------|
|
|
||||||
| AWX endpoint | **Configured** | "lab-awx" at http://10.10.0.122:30080, created 2026-03-03 11:17:36 |
|
|
||||||
| AWX cluster config | **Configured** | hz-cluster linked to lab-awx, post-deploy template 9, decommission template 10, 600s timeout |
|
|
||||||
| AWX health | **Healthy** | Version 24.6.1, reachable |
|
|
||||||
|
|
||||||
## Observations
|
|
||||||
|
|
||||||
### Cluster ID changed (52 → 53)
|
|
||||||
|
|
||||||
The previous Aether instance had the cluster registered as ID 52 (visible
|
|
||||||
in haproxy.yaml and awx.yaml configs). After redeploy, the new database
|
|
||||||
assigned it ID 53. This is a synthetic auto-increment ID internal to
|
|
||||||
Aether's DB. The cluster itself (certificate, nodes, instances) is
|
|
||||||
unchanged — only Aether's internal reference number differs.
|
|
||||||
|
|
||||||
**Impact**: Scripts that hardcode `cluster_id: 52` (haproxy.yaml, awx.yaml)
|
|
||||||
would need updating if they interact with Aether's API using this ID.
|
|
||||||
The deploy scripts use the cluster_id for Aether API calls (HAProxy image
|
|
||||||
push, AWX cluster config). If these scripts are re-run against this Aether
|
|
||||||
instance, the ID mismatch would cause failures.
|
|
||||||
|
|
||||||
### HAProxy image survived, push state didn't
|
|
||||||
|
|
||||||
The HAProxy base image v2.64 (287 MB, built 2026-02-21) is stored on
|
|
||||||
Aether's local filesystem, not in the DB. It survived the redeploy. But
|
|
||||||
the DB record of which clusters it was pushed to was lost. The UI shows
|
|
||||||
"No Image" for hz-cluster, meaning the image needs to be re-pushed before
|
|
||||||
HAProxy LB management works through Aether.
|
|
||||||
|
|
||||||
The HAProxy containers themselves (ffsdn-haproxy-52-01/02) are running
|
|
||||||
fine in the cluster — they don't depend on Aether for runtime operation.
|
|
||||||
Aether only manages their configuration and deployment lifecycle.
|
|
||||||
|
|
||||||
### Settings are all defaults
|
|
||||||
|
|
||||||
All settings reverted to defaults. Notable:
|
|
||||||
- `log_level: Debug` (default, verbose — consider changing to Info)
|
|
||||||
- `refresh_interval_minutes: 5`
|
|
||||||
- `auto_backup_enabled: False` — **should enable this time**
|
|
||||||
- `manage_instances: True`
|
|
||||||
- `manage_vcenters: False`
|
|
||||||
- `migration_manager: False`
|
|
||||||
|
|
||||||
### What this tells us about Aether's architecture
|
|
||||||
|
|
||||||
1. **Cluster state is ephemeral in Aether** — it's a cache of the Incus API.
|
|
||||||
Losing the DB doesn't lose infrastructure visibility. Re-add the cluster
|
|
||||||
and everything reappears within one sync cycle (5 min default).
|
|
||||||
|
|
||||||
2. **Operational config is DB-only** — blueprints, ACL rules, RBAC, AWX
|
|
||||||
endpoints, OC connections, and deployment history live exclusively in
|
|
||||||
PostgreSQL. No export/import mechanism was observed.
|
|
||||||
|
|
||||||
3. **The license is tied to the DB** — reinstalling generates a new
|
|
||||||
activation code, invalidating any existing license key. This is
|
|
||||||
explicitly noted on the licensing page.
|
|
||||||
|
|
||||||
4. **HAProxy images are filesystem-based** — they survive DB loss but
|
|
||||||
lose their cluster push state. The containers in the cluster are
|
|
||||||
independent of Aether once deployed.
|
|
||||||
|
|
||||||
5. **AWX integration is lightweight** — just an endpoint URL + template IDs.
|
|
||||||
Quick to re-add manually (as was done). Job history is lost but AWX
|
|
||||||
itself retains its own job history.
|
|
||||||
|
|
||||||
## Recovery priority if this happens again
|
|
||||||
|
|
||||||
1. Enable `auto_backup_enabled` immediately after setup
|
|
||||||
2. Periodically copy `/opt/ffsdn/backups/` off the Aether VM
|
|
||||||
3. After restore: re-add cluster, re-add AWX endpoint, re-push HAProxy image
|
|
||||||
4. Blueprints and ACL rules would need to be recreated from scratch
|
|
||||||
(consider documenting them externally)
|
|
||||||
Loading…
Reference in New Issue