---
paths:
  - "bin/deploy-haproxy"
  - "bin/helpers/aether-browser"
  - "notes/haproxy-guide.md"
---

# HAProxy Load Balancing (Aether-Managed)

## Overview

- Aether manages HAProxy deployment: image builds, HA pair, VIP management,
  L7 service configuration, health monitoring.
- Lab cluster: oc-lab-cluster (ID: 52), OVN network: net-prod (10.10.10.0/24).
- VIP range: 192.168.103.200-210 (from UPLINK).
- `deploy-haproxy` script: `--deploy`, `--status`, `--cleanup`, `--doctor`.

## Use Playwright for Aether session auth, NOT curl

HAProxy management endpoints use session-authenticated routes with CSRF
protection. **curl cannot reliably handle this** — use Playwright instead.

- **Playwright MCP**: `.mcp.json` configures `@playwright/mcp@latest`.
  When loaded, use `browser_navigate`, `browser_click`, `browser_fill` tools.
- **Helper**: `incusos/helpers/aether-browser` for standalone automation.
- **Login page**: `GET /` (root, NOT `/login`). Form action: `POST /login`.
- **CSRF token**: hidden `<input name="csrf_token">` in HTML form, NOT cookie.

## Key UI element IDs (for Playwright automation)

| Element | ID | Purpose |
|---------|-----|---------|
| Cluster dropdown | `#clusterSelect` | Select cluster to manage |
| Deploy mode | `#deployMode` | OVN or Keepalived |
| OVN network | `#deployNetwork` | Select network (populates IP dropdowns) |
| LB VIP select | `#deployLBVIPSelect` | Choose VIP from UPLINK range |
| HP01 IP select | `#deployHP01IPSelect` | Choose HAProxy 01 IP |
| HP02 IP select | `#deployHP02IPSelect` | Choose HAProxy 02 IP |
| CPU limit | `#deployCPU` | Default: 2 |
| Memory limit | `#deployMemory` | Default: 1 GB |
| Service VIP | `#serviceVIP` | VIP by ID (e.g., value="41") |
| Service mode | `#serviceMode` | tcp, http, https-passthrough, https-termination |
| Service port | `#servicePort` | Listen port |
| Service balance | `#serviceBalance` | roundrobin, leastconn, source, first |
| Health check | `#serviceHealthCheck` | Checkbox, default: checked |
| Backend name | `input[name="backend_name_N"]` | Nth backend name |
| Backend IP | `input[name="backend_ip_N"]` | Nth backend IP |
| Backend port | `input[name="backend_port_N"]` | Nth backend port |

Key JS functions: `showEditServiceModal(serviceId)`,
`showServiceHealthStats(serviceId, name)`, `reloadHAProxyConfig()`,
`showAddServiceModal()`, `showAddVIPModalWithVIPs()`.

## Key API routes (session-authenticated, use via Playwright page.evaluate)

| Route | Method | Purpose |
|-------|--------|---------|
| `/haproxy/infrastructure/{cluster_id}` | GET | Deployment state (JSON) |
| `/haproxy/infrastructure/deploy` | POST | Deploy infrastructure |
| `/haproxy/service` | POST | Create service |
| `/haproxy/service/{id}` | PUT | Update service |
| `/haproxy/service/{id}` | DELETE | Delete service |
| `/haproxy/reload/{cluster_id}` | POST | Reload HAProxy config |
| `/haproxy/infrastructure/{cluster_id}` | DELETE | Remove infrastructure |

## API field names (validated 2026-02-24)

**Deploy infrastructure** (`POST /haproxy/infrastructure/deploy`):
`cluster_id` (int), `ovn_network`, `lb_vip` (NOT `vip`), `haproxy_01_ip`,
`haproxy_02_ip`, `cpu_limit` (string), `memory_limit` (string, e.g. "1GB")

**Create service** (`POST /haproxy/service`):
`vip_id` (int, NOT VIP address), `name`, `description`, `hostname`,
`listen_port` (int), `mode`, `balance_method`, `health_check_enabled` (bool),
`session_persistence` (bool), `backends` array with objects:
`{ name, target_ip, target_port (int), weight (int) }`

## Container naming

`ffsdn-haproxy-{clusterID}-01` and `ffsdn-haproxy-{clusterID}-02`
For cluster 52: `ffsdn-haproxy-52-01`, `ffsdn-haproxy-52-02`.

## Lab IP assignments (live, validated 2026-02-24)

| Component | IP |
|-----------|-----|
| VIP | 192.168.103.200 |
| HAProxy 01 | 10.10.10.50 (oc-node-01) |
| HAProxy 02 | 10.10.10.51 (oc-node-02) |
| nginx-lb-01 | 10.10.10.60 |
| nginx-lb-02 | 10.10.10.61 |
| nginx-lb-03 | 10.10.10.62 |

## Validated gotchas

- **OVN LB not created by Aether**: Aether deploys HAProxy containers and
  ACLs but does NOT create the OVN load balancer (`incus network load-balancer`).
  You must create it manually after deployment. See guide Step 3.
- **HTTP mode health checks**: Aether generates `option httpchk` (sends
  `OPTIONS /`). Nginx returns 405 → backends marked DOWN. **Use TCP mode**
  for nginx backends, or configure nginx to accept OPTIONS.
- **HP-01 may not auto-start**: After deploy, `ffsdn-haproxy-52-01` was
  sometimes found Stopped. Check both containers post-deploy. Not always
  reproducible — both started on the 2026-02-24 re-validation.
- **OVN VIP external access**: Aether ACLs only allow ingress from
  `source: VIP_IP`. External traffic preserves original client IP → rejected.
  **Fix**: add broad ingress rule to each HAProxy ACL:
  `incus network acl rule add <remote>:<acl> ingress action=allow protocol=tcp destination_port=80,443`
  Must re-apply after infrastructure redeploy.
- **Internal VIP not reachable**: The manually-created OVN LB handles
  external-to-internal traffic but does NOT hairpin for internal-to-VIP
  requests from the same logical switch. Use HAProxy localhost for internal tests.
- **OVN LB has no health checking**: When one HAProxy instance is stopped,
  the OVN LB continues routing ~50% of connections to the dead backend,
  causing timeouts. Failover is not instant. Consider Keepalived mode for
  production deployments requiring true active/standby failover.
- **Static IPs on OVN**: Use `ipv4.address` device key on the NIC device,
  not systemd-networkd: `incus launch ... -d eth0,ipv4.address=10.10.10.60`.

## Key differences from deploy-awx

- **No VM creation**: HAProxy uses Aether-managed containers, not a standalone VM.
- **Playwright auth**: Uses browser automation, not JWT or basic auth.
- **No --heal action**: HAProxy is fully Aether-managed. Recovery = `--cleanup` + `--deploy`.
- **Backends via incus CLI**: Test nginx backends created directly with `ipv4.address` device key.