diff --git a/.claude/rules/haproxy-lb.md b/.claude/rules/haproxy-lb.md new file mode 100644 index 0000000..d5efa5e --- /dev/null +++ b/.claude/rules/haproxy-lb.md @@ -0,0 +1,125 @@ +--- +paths: + - "incusos/deploy-haproxy" + - "incusos/helpers/aether-browser" + - "notes/haproxy-guide.md" +--- + +# HAProxy Load Balancing (Aether-Managed) + +## Overview + +- Aether manages HAProxy deployment: image builds, HA pair, VIP management, + L7 service configuration, health monitoring. +- Lab cluster: oc-lab-cluster (ID: 52), OVN network: net-prod (10.10.10.0/24). +- VIP range: 192.168.103.200-210 (from UPLINK). +- `deploy-haproxy` script: `--deploy`, `--status`, `--cleanup`, `--doctor`. + +## Use Playwright for Aether session auth, NOT curl + +HAProxy management endpoints use session-authenticated routes with CSRF +protection. **curl cannot reliably handle this** — use Playwright instead. + +- **Playwright MCP**: `.mcp.json` configures `@playwright/mcp@latest`. + When loaded, use `browser_navigate`, `browser_click`, `browser_fill` tools. +- **Helper**: `incusos/helpers/aether-browser` for standalone automation. +- **Login page**: `GET /` (root, NOT `/login`). Form action: `POST /login`. +- **CSRF token**: hidden `` in HTML form, NOT cookie. + +## Key UI element IDs (for Playwright automation) + +| Element | ID | Purpose | +|---------|-----|---------| +| Cluster dropdown | `#clusterSelect` | Select cluster to manage | +| Deploy mode | `#deployMode` | OVN or Keepalived | +| OVN network | `#deployNetwork` | Select network (populates IP dropdowns) | +| LB VIP select | `#deployLBVIPSelect` | Choose VIP from UPLINK range | +| HP01 IP select | `#deployHP01IPSelect` | Choose HAProxy 01 IP | +| HP02 IP select | `#deployHP02IPSelect` | Choose HAProxy 02 IP | +| CPU limit | `#deployCPU` | Default: 2 | +| Memory limit | `#deployMemory` | Default: 1 GB | +| Service VIP | `#serviceVIP` | VIP by ID (e.g., value="41") | +| Service mode | `#serviceMode` | tcp, http, https-passthrough, https-termination | +| Service port | `#servicePort` | Listen port | +| Service balance | `#serviceBalance` | roundrobin, leastconn, source, first | +| Health check | `#serviceHealthCheck` | Checkbox, default: checked | +| Backend name | `input[name="backend_name_N"]` | Nth backend name | +| Backend IP | `input[name="backend_ip_N"]` | Nth backend IP | +| Backend port | `input[name="backend_port_N"]` | Nth backend port | + +Key JS functions: `showEditServiceModal(serviceId)`, +`showServiceHealthStats(serviceId, name)`, `reloadHAProxyConfig()`, +`showAddServiceModal()`, `showAddVIPModalWithVIPs()`. + +## Key API routes (session-authenticated, use via Playwright page.evaluate) + +| Route | Method | Purpose | +|-------|--------|---------| +| `/haproxy/infrastructure/{cluster_id}` | GET | Deployment state (JSON) | +| `/haproxy/infrastructure/deploy` | POST | Deploy infrastructure | +| `/haproxy/service` | POST | Create service | +| `/haproxy/service/{id}` | PUT | Update service | +| `/haproxy/service/{id}` | DELETE | Delete service | +| `/haproxy/reload/{cluster_id}` | POST | Reload HAProxy config | +| `/haproxy/infrastructure/{cluster_id}` | DELETE | Remove infrastructure | + +## API field names (validated 2026-02-24) + +**Deploy infrastructure** (`POST /haproxy/infrastructure/deploy`): +`cluster_id` (int), `ovn_network`, `lb_vip` (NOT `vip`), `haproxy_01_ip`, +`haproxy_02_ip`, `cpu_limit` (string), `memory_limit` (string, e.g. "1GB") + +**Create service** (`POST /haproxy/service`): +`vip_id` (int, NOT VIP address), `name`, `description`, `hostname`, +`listen_port` (int), `mode`, `balance_method`, `health_check_enabled` (bool), +`session_persistence` (bool), `backends` array with objects: +`{ name, target_ip, target_port (int), weight (int) }` + +## Container naming + +`ffsdn-haproxy-{clusterID}-01` and `ffsdn-haproxy-{clusterID}-02` +For cluster 52: `ffsdn-haproxy-52-01`, `ffsdn-haproxy-52-02`. + +## Lab IP assignments (live, validated 2026-02-24) + +| Component | IP | +|-----------|-----| +| VIP | 192.168.103.200 | +| HAProxy 01 | 10.10.10.50 (oc-node-01) | +| HAProxy 02 | 10.10.10.51 (oc-node-02) | +| nginx-lb-01 | 10.10.10.60 | +| nginx-lb-02 | 10.10.10.61 | +| nginx-lb-03 | 10.10.10.62 | + +## Validated gotchas + +- **OVN LB not created by Aether**: Aether deploys HAProxy containers and + ACLs but does NOT create the OVN load balancer (`incus network load-balancer`). + You must create it manually after deployment. See guide Step 3. +- **HTTP mode health checks**: Aether generates `option httpchk` (sends + `OPTIONS /`). Nginx returns 405 → backends marked DOWN. **Use TCP mode** + for nginx backends, or configure nginx to accept OPTIONS. +- **HP-01 may not auto-start**: After deploy, `ffsdn-haproxy-52-01` was + sometimes found Stopped. Check both containers post-deploy. Not always + reproducible — both started on the 2026-02-24 re-validation. +- **OVN VIP external access**: Aether ACLs only allow ingress from + `source: VIP_IP`. External traffic preserves original client IP → rejected. + **Fix**: add broad ingress rule to each HAProxy ACL: + `incus network acl rule add : ingress action=allow protocol=tcp destination_port=80,443` + Must re-apply after infrastructure redeploy. +- **Internal VIP not reachable**: The manually-created OVN LB handles + external-to-internal traffic but does NOT hairpin for internal-to-VIP + requests from the same logical switch. Use HAProxy localhost for internal tests. +- **OVN LB has no health checking**: When one HAProxy instance is stopped, + the OVN LB continues routing ~50% of connections to the dead backend, + causing timeouts. Failover is not instant. Consider Keepalived mode for + production deployments requiring true active/standby failover. +- **Static IPs on OVN**: Use `ipv4.address` device key on the NIC device, + not systemd-networkd: `incus launch ... -d eth0,ipv4.address=10.10.10.60`. + +## Key differences from deploy-awx + +- **No VM creation**: HAProxy uses Aether-managed containers, not a standalone VM. +- **Playwright auth**: Uses browser automation, not JWT or basic auth. +- **No --heal action**: HAProxy is fully Aether-managed. Recovery = `--cleanup` + `--deploy`. +- **Backends via incus CLI**: Test nginx backends created directly with `ipv4.address` device key. diff --git a/.mcp.json b/.mcp.json new file mode 100644 index 0000000..259a959 --- /dev/null +++ b/.mcp.json @@ -0,0 +1,8 @@ +{ + "mcpServers": { + "playwright": { + "command": "npx", + "args": ["@playwright/mcp@latest"] + } + } +} diff --git a/CLAUDE.md b/CLAUDE.md index 8b12ac0..cb7d47d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -26,10 +26,12 @@ incus-contrib/ │ ├── incusos-seed # Seed archive generator (Linux + macOS) │ ├── incusos-proxmox # Declarative Proxmox VM deployment + lab lifecycle │ ├── deploy-awx # AWX deployment + management on Incus cluster +│ ├── deploy-haproxy # HAProxy LB deployment + management via Aether │ ├── awx-manifests/ # K8s manifests for AWX Operator + instance │ ├── helpers/ │ │ ├── proxmox-screenshot # VMID -> PNG console screenshot -│ │ └── proxmox-api # Authenticated API calls (handles ! in token) +│ │ ├── proxmox-api # Authenticated API calls (handles ! in token) +│ │ └── aether-browser # Playwright browser automation for Aether web UI │ ├── lab-test # Guided lab validation (12 test phases) │ ├── observe-deploy # Single-VM deploy with console screenshots │ ├── proxmox.yaml # Proxmox connection (gitignored) @@ -95,6 +97,42 @@ curl -sk http://192.168.102.161:30080/api/v2/jobs/{JOB_ID}/stdout/?format=txt \ -H "Authorization: Bearer $AWX_TOKEN" ``` +### Aether browser automation (Playwright) + +Many Aether features (HAProxy management, blueprints, deploys) are **not in +the JWT API** — they use session-authenticated routes with CSRF protection +that curl cannot handle reliably. **USE PLAYWRIGHT** for these interactions. + +**MCP server**: Configured in `.mcp.json` — provides `browser_navigate`, +`browser_click`, `browser_fill`, `browser_screenshot` etc. as tools when +the Playwright MCP server is running. Prefer MCP tools when available. + +**Helper script**: `incusos/helpers/aether-browser` provides standalone +Playwright automation when MCP tools aren't loaded: +```bash +source env # loads AETHER_ADMIN_PASSWORD +NODE_PATH=~/node_modules node incusos/helpers/aether-browser [args] +``` + +Actions: +- `login` — authenticate and save session cookies to `/tmp/aether-cookies.json` +- `screenshot ` — navigate to path, take screenshot +- `navigate ` — navigate and show page title/URL +- `haproxy-status` — screenshot HAProxy management page +- `haproxy-images` — list built HAProxy images via session API +- `api-get ` — authenticated GET using browser session +- `api-post ` — authenticated POST with CSRF handling +- `eval ` — evaluate JavaScript on the current page + +**When to use what**: +- JWT API (`/api/*` endpoints): use `curl` with bearer token +- Session-authenticated routes (`/haproxy/*`, UI actions): use Playwright +- Never use curl for session auth — CSRF protection requires browser cookies + +**Key technical detail**: Aether login is at `/` (root), NOT `/login`. +The form POSTs to `/login`. CSRF token is in a hidden `` field, +not in cookies. + ## Critical safety rules - **Proxmox SSH is screenshots-only.** Full rules in `.claude/rules/proxmox-ssh-rules.md`. @@ -146,6 +184,7 @@ which files are being edited: | `operations-center.md` | OC guide, incusos-proxmox, OC example configs | | `networking-storage.md` | networking, storage, migration, UTM guides | | `awx-integration.md` | ansible/, deploy-awx, awx-manifests, AWX guide | +| `haproxy-lb.md` | deploy-haproxy, HAProxy guide | | `proxmox-ssh-rules.md` | **Always loaded** (no paths filter) | | `lab-infrastructure.md` | incusos-proxmox, lab-test, examples | diff --git a/incusos/deploy-haproxy b/incusos/deploy-haproxy new file mode 100755 index 0000000..916d508 --- /dev/null +++ b/incusos/deploy-haproxy @@ -0,0 +1,1211 @@ +#!/usr/bin/env bash +# deploy-haproxy - Deploy and manage HAProxy load balancing on an Incus cluster +# +# Deploys test backends (nginx), builds a HAProxy image via Aether, +# deploys HA infrastructure (2x HAProxy + OVN LB), and creates an +# HTTP service with health checks. +# +# Uses Aether session-based authentication (not JWT) for HAProxy +# management endpoints that are not in the public API. +# +# Usage: deploy-haproxy [OPTIONS] +# Run 'deploy-haproxy --help' for full usage information. + +set -euo pipefail + +readonly VERSION="0.1.0" +readonly SCRIPT_NAME="deploy-haproxy" +readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +readonly ENV_FILE="${SCRIPT_DIR}/../env" + +# --------------------------------------------------------------------------- +# Defaults (overridden by config file or CLI flags) +# --------------------------------------------------------------------------- + +CLUSTER_REMOTE="oc-node-01" +CLUSTER_ID="52" +OVN_NETWORK="net-prod" +VIP="192.168.103.200" +HAPROXY_01_IP="10.10.10.50" +HAPROXY_02_IP="10.10.10.51" +HAPROXY_CPU="2" +HAPROXY_MEMORY="1GB" +BACKEND_IPS="10.10.10.60,10.10.10.61,10.10.10.62" +BACKEND_PORT="80" +SERVICE_NAME="web-test" +SERVICE_MODE="http" +SERVICE_BALANCE="roundrobin" +IMAGE_VERSION="1.0.0" +AETHER_URL="https://192.168.102.160:8443" +AETHER_USER="admin" +AETHER_PASSWORD="" + +# Runtime flags +DRY_RUN=false +VERBOSE=false +QUIET=false +CONFIG_FILE="" +ACTION="" +SKIP_BACKENDS=false +SKIP_IMAGE=false + +# Session state +COOKIE_JAR="" +CSRF_TOKEN="" + +# --------------------------------------------------------------------------- +# Color and formatting +# --------------------------------------------------------------------------- + +setup_colors() { + if [[ -t 1 ]] && [[ "${NO_COLOR:-}" != "1" ]] && [[ "${TERM:-}" != "dumb" ]]; then + RED='\033[0;31m' + GREEN='\033[0;32m' + YELLOW='\033[0;33m' + BLUE='\033[0;34m' + CYAN='\033[0;36m' + BOLD='\033[1m' + DIM='\033[2m' + RESET='\033[0m' + else + RED='' GREEN='' YELLOW='' BLUE='' CYAN='' BOLD='' DIM='' RESET='' + fi +} + +info() { [[ "$QUIET" == true ]] && return; echo -e "${BLUE}[info]${RESET} $*"; } +success() { [[ "$QUIET" == true ]] && return; echo -e "${GREEN}[ok]${RESET} $*"; } +warn() { echo -e "${YELLOW}[warn]${RESET} $*" >&2; } +error() { echo -e "${RED}[error]${RESET} $*" >&2; } +step() { [[ "$QUIET" == true ]] && return; echo -e "${CYAN}[step]${RESET} ${BOLD}$*${RESET}"; } +detail() { [[ "$VERBOSE" != true ]] && return; echo -e "${DIM} $*${RESET}"; } + +dry_run_guard() { + if [[ "$DRY_RUN" == true ]]; then + info "(dry-run) $*" + return 1 + fi + return 0 +} + +# --------------------------------------------------------------------------- +# Help +# --------------------------------------------------------------------------- + +usage() { + cat <