126 lines
5.9 KiB
Markdown
126 lines
5.9 KiB
Markdown
---
|
|
paths:
|
|
- "incusos/deploy-haproxy"
|
|
- "incusos/helpers/aether-browser"
|
|
- "notes/haproxy-guide.md"
|
|
---
|
|
|
|
# HAProxy Load Balancing (Aether-Managed)
|
|
|
|
## Overview
|
|
|
|
- Aether manages HAProxy deployment: image builds, HA pair, VIP management,
|
|
L7 service configuration, health monitoring.
|
|
- Lab cluster: oc-lab-cluster (ID: 52), OVN network: net-prod (10.10.10.0/24).
|
|
- VIP range: 192.168.103.200-210 (from UPLINK).
|
|
- `deploy-haproxy` script: `--deploy`, `--status`, `--cleanup`, `--doctor`.
|
|
|
|
## Use Playwright for Aether session auth, NOT curl
|
|
|
|
HAProxy management endpoints use session-authenticated routes with CSRF
|
|
protection. **curl cannot reliably handle this** — use Playwright instead.
|
|
|
|
- **Playwright MCP**: `.mcp.json` configures `@playwright/mcp@latest`.
|
|
When loaded, use `browser_navigate`, `browser_click`, `browser_fill` tools.
|
|
- **Helper**: `incusos/helpers/aether-browser` for standalone automation.
|
|
- **Login page**: `GET /` (root, NOT `/login`). Form action: `POST /login`.
|
|
- **CSRF token**: hidden `<input name="csrf_token">` in HTML form, NOT cookie.
|
|
|
|
## Key UI element IDs (for Playwright automation)
|
|
|
|
| Element | ID | Purpose |
|
|
|---------|-----|---------|
|
|
| Cluster dropdown | `#clusterSelect` | Select cluster to manage |
|
|
| Deploy mode | `#deployMode` | OVN or Keepalived |
|
|
| OVN network | `#deployNetwork` | Select network (populates IP dropdowns) |
|
|
| LB VIP select | `#deployLBVIPSelect` | Choose VIP from UPLINK range |
|
|
| HP01 IP select | `#deployHP01IPSelect` | Choose HAProxy 01 IP |
|
|
| HP02 IP select | `#deployHP02IPSelect` | Choose HAProxy 02 IP |
|
|
| CPU limit | `#deployCPU` | Default: 2 |
|
|
| Memory limit | `#deployMemory` | Default: 1 GB |
|
|
| Service VIP | `#serviceVIP` | VIP by ID (e.g., value="41") |
|
|
| Service mode | `#serviceMode` | tcp, http, https-passthrough, https-termination |
|
|
| Service port | `#servicePort` | Listen port |
|
|
| Service balance | `#serviceBalance` | roundrobin, leastconn, source, first |
|
|
| Health check | `#serviceHealthCheck` | Checkbox, default: checked |
|
|
| Backend name | `input[name="backend_name_N"]` | Nth backend name |
|
|
| Backend IP | `input[name="backend_ip_N"]` | Nth backend IP |
|
|
| Backend port | `input[name="backend_port_N"]` | Nth backend port |
|
|
|
|
Key JS functions: `showEditServiceModal(serviceId)`,
|
|
`showServiceHealthStats(serviceId, name)`, `reloadHAProxyConfig()`,
|
|
`showAddServiceModal()`, `showAddVIPModalWithVIPs()`.
|
|
|
|
## Key API routes (session-authenticated, use via Playwright page.evaluate)
|
|
|
|
| Route | Method | Purpose |
|
|
|-------|--------|---------|
|
|
| `/haproxy/infrastructure/{cluster_id}` | GET | Deployment state (JSON) |
|
|
| `/haproxy/infrastructure/deploy` | POST | Deploy infrastructure |
|
|
| `/haproxy/service` | POST | Create service |
|
|
| `/haproxy/service/{id}` | PUT | Update service |
|
|
| `/haproxy/service/{id}` | DELETE | Delete service |
|
|
| `/haproxy/reload/{cluster_id}` | POST | Reload HAProxy config |
|
|
| `/haproxy/infrastructure/{cluster_id}` | DELETE | Remove infrastructure |
|
|
|
|
## API field names (validated 2026-02-24)
|
|
|
|
**Deploy infrastructure** (`POST /haproxy/infrastructure/deploy`):
|
|
`cluster_id` (int), `ovn_network`, `lb_vip` (NOT `vip`), `haproxy_01_ip`,
|
|
`haproxy_02_ip`, `cpu_limit` (string), `memory_limit` (string, e.g. "1GB")
|
|
|
|
**Create service** (`POST /haproxy/service`):
|
|
`vip_id` (int, NOT VIP address), `name`, `description`, `hostname`,
|
|
`listen_port` (int), `mode`, `balance_method`, `health_check_enabled` (bool),
|
|
`session_persistence` (bool), `backends` array with objects:
|
|
`{ name, target_ip, target_port (int), weight (int) }`
|
|
|
|
## Container naming
|
|
|
|
`ffsdn-haproxy-{clusterID}-01` and `ffsdn-haproxy-{clusterID}-02`
|
|
For cluster 52: `ffsdn-haproxy-52-01`, `ffsdn-haproxy-52-02`.
|
|
|
|
## Lab IP assignments (live, validated 2026-02-24)
|
|
|
|
| Component | IP |
|
|
|-----------|-----|
|
|
| VIP | 192.168.103.200 |
|
|
| HAProxy 01 | 10.10.10.50 (oc-node-01) |
|
|
| HAProxy 02 | 10.10.10.51 (oc-node-02) |
|
|
| nginx-lb-01 | 10.10.10.60 |
|
|
| nginx-lb-02 | 10.10.10.61 |
|
|
| nginx-lb-03 | 10.10.10.62 |
|
|
|
|
## Validated gotchas
|
|
|
|
- **OVN LB not created by Aether**: Aether deploys HAProxy containers and
|
|
ACLs but does NOT create the OVN load balancer (`incus network load-balancer`).
|
|
You must create it manually after deployment. See guide Step 3.
|
|
- **HTTP mode health checks**: Aether generates `option httpchk` (sends
|
|
`OPTIONS /`). Nginx returns 405 → backends marked DOWN. **Use TCP mode**
|
|
for nginx backends, or configure nginx to accept OPTIONS.
|
|
- **HP-01 may not auto-start**: After deploy, `ffsdn-haproxy-52-01` was
|
|
sometimes found Stopped. Check both containers post-deploy. Not always
|
|
reproducible — both started on the 2026-02-24 re-validation.
|
|
- **OVN VIP external access**: Aether ACLs only allow ingress from
|
|
`source: VIP_IP`. External traffic preserves original client IP → rejected.
|
|
**Fix**: add broad ingress rule to each HAProxy ACL:
|
|
`incus network acl rule add <remote>:<acl> ingress action=allow protocol=tcp destination_port=80,443`
|
|
Must re-apply after infrastructure redeploy.
|
|
- **Internal VIP not reachable**: The manually-created OVN LB handles
|
|
external-to-internal traffic but does NOT hairpin for internal-to-VIP
|
|
requests from the same logical switch. Use HAProxy localhost for internal tests.
|
|
- **OVN LB has no health checking**: When one HAProxy instance is stopped,
|
|
the OVN LB continues routing ~50% of connections to the dead backend,
|
|
causing timeouts. Failover is not instant. Consider Keepalived mode for
|
|
production deployments requiring true active/standby failover.
|
|
- **Static IPs on OVN**: Use `ipv4.address` device key on the NIC device,
|
|
not systemd-networkd: `incus launch ... -d eth0,ipv4.address=10.10.10.60`.
|
|
|
|
## Key differences from deploy-awx
|
|
|
|
- **No VM creation**: HAProxy uses Aether-managed containers, not a standalone VM.
|
|
- **Playwright auth**: Uses browser automation, not JWT or basic auth.
|
|
- **No --heal action**: HAProxy is fully Aether-managed. Recovery = `--cleanup` + `--deploy`.
|
|
- **Backends via incus CLI**: Test nginx backends created directly with `ipv4.address` device key.
|