Add HAProxy LB guide, deploy script, Playwright helper; validate end-to-end
Tear down and fully re-deploy HAProxy infrastructure from scratch to validate the guide reads correctly. Key fixes from re-validation: - Fix session API field names: lb_vip (not vip), target_ip/target_port (not ip/port), health_check_enabled (not health_check) - Document that Aether does NOT create the OVN load balancer — must be created manually via incus network load-balancer - Fix internal VIP test: OVN LB doesn't hairpin, use localhost instead - Document OVN LB has no health checking (HAProxy failover not instant) - Add cleanup step for manually-created OVN LB - Add aether-browser Playwright helper and .mcp.json config - Add deploy-haproxy script, haproxy-lb rules, full guide Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
0100182dd8
commit
c047bc6ed4
|
|
@ -0,0 +1,125 @@
|
||||||
|
---
|
||||||
|
paths:
|
||||||
|
- "incusos/deploy-haproxy"
|
||||||
|
- "incusos/helpers/aether-browser"
|
||||||
|
- "notes/haproxy-guide.md"
|
||||||
|
---
|
||||||
|
|
||||||
|
# HAProxy Load Balancing (Aether-Managed)
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
- Aether manages HAProxy deployment: image builds, HA pair, VIP management,
|
||||||
|
L7 service configuration, health monitoring.
|
||||||
|
- Lab cluster: oc-lab-cluster (ID: 52), OVN network: net-prod (10.10.10.0/24).
|
||||||
|
- VIP range: 192.168.103.200-210 (from UPLINK).
|
||||||
|
- `deploy-haproxy` script: `--deploy`, `--status`, `--cleanup`, `--doctor`.
|
||||||
|
|
||||||
|
## Use Playwright for Aether session auth, NOT curl
|
||||||
|
|
||||||
|
HAProxy management endpoints use session-authenticated routes with CSRF
|
||||||
|
protection. **curl cannot reliably handle this** — use Playwright instead.
|
||||||
|
|
||||||
|
- **Playwright MCP**: `.mcp.json` configures `@playwright/mcp@latest`.
|
||||||
|
When loaded, use `browser_navigate`, `browser_click`, `browser_fill` tools.
|
||||||
|
- **Helper**: `incusos/helpers/aether-browser` for standalone automation.
|
||||||
|
- **Login page**: `GET /` (root, NOT `/login`). Form action: `POST /login`.
|
||||||
|
- **CSRF token**: hidden `<input name="csrf_token">` in HTML form, NOT cookie.
|
||||||
|
|
||||||
|
## Key UI element IDs (for Playwright automation)
|
||||||
|
|
||||||
|
| Element | ID | Purpose |
|
||||||
|
|---------|-----|---------|
|
||||||
|
| Cluster dropdown | `#clusterSelect` | Select cluster to manage |
|
||||||
|
| Deploy mode | `#deployMode` | OVN or Keepalived |
|
||||||
|
| OVN network | `#deployNetwork` | Select network (populates IP dropdowns) |
|
||||||
|
| LB VIP select | `#deployLBVIPSelect` | Choose VIP from UPLINK range |
|
||||||
|
| HP01 IP select | `#deployHP01IPSelect` | Choose HAProxy 01 IP |
|
||||||
|
| HP02 IP select | `#deployHP02IPSelect` | Choose HAProxy 02 IP |
|
||||||
|
| CPU limit | `#deployCPU` | Default: 2 |
|
||||||
|
| Memory limit | `#deployMemory` | Default: 1 GB |
|
||||||
|
| Service VIP | `#serviceVIP` | VIP by ID (e.g., value="41") |
|
||||||
|
| Service mode | `#serviceMode` | tcp, http, https-passthrough, https-termination |
|
||||||
|
| Service port | `#servicePort` | Listen port |
|
||||||
|
| Service balance | `#serviceBalance` | roundrobin, leastconn, source, first |
|
||||||
|
| Health check | `#serviceHealthCheck` | Checkbox, default: checked |
|
||||||
|
| Backend name | `input[name="backend_name_N"]` | Nth backend name |
|
||||||
|
| Backend IP | `input[name="backend_ip_N"]` | Nth backend IP |
|
||||||
|
| Backend port | `input[name="backend_port_N"]` | Nth backend port |
|
||||||
|
|
||||||
|
Key JS functions: `showEditServiceModal(serviceId)`,
|
||||||
|
`showServiceHealthStats(serviceId, name)`, `reloadHAProxyConfig()`,
|
||||||
|
`showAddServiceModal()`, `showAddVIPModalWithVIPs()`.
|
||||||
|
|
||||||
|
## Key API routes (session-authenticated, use via Playwright page.evaluate)
|
||||||
|
|
||||||
|
| Route | Method | Purpose |
|
||||||
|
|-------|--------|---------|
|
||||||
|
| `/haproxy/infrastructure/{cluster_id}` | GET | Deployment state (JSON) |
|
||||||
|
| `/haproxy/infrastructure/deploy` | POST | Deploy infrastructure |
|
||||||
|
| `/haproxy/service` | POST | Create service |
|
||||||
|
| `/haproxy/service/{id}` | PUT | Update service |
|
||||||
|
| `/haproxy/service/{id}` | DELETE | Delete service |
|
||||||
|
| `/haproxy/reload/{cluster_id}` | POST | Reload HAProxy config |
|
||||||
|
| `/haproxy/infrastructure/{cluster_id}` | DELETE | Remove infrastructure |
|
||||||
|
|
||||||
|
## API field names (validated 2026-02-24)
|
||||||
|
|
||||||
|
**Deploy infrastructure** (`POST /haproxy/infrastructure/deploy`):
|
||||||
|
`cluster_id` (int), `ovn_network`, `lb_vip` (NOT `vip`), `haproxy_01_ip`,
|
||||||
|
`haproxy_02_ip`, `cpu_limit` (string), `memory_limit` (string, e.g. "1GB")
|
||||||
|
|
||||||
|
**Create service** (`POST /haproxy/service`):
|
||||||
|
`vip_id` (int, NOT VIP address), `name`, `description`, `hostname`,
|
||||||
|
`listen_port` (int), `mode`, `balance_method`, `health_check_enabled` (bool),
|
||||||
|
`session_persistence` (bool), `backends` array with objects:
|
||||||
|
`{ name, target_ip, target_port (int), weight (int) }`
|
||||||
|
|
||||||
|
## Container naming
|
||||||
|
|
||||||
|
`ffsdn-haproxy-{clusterID}-01` and `ffsdn-haproxy-{clusterID}-02`
|
||||||
|
For cluster 52: `ffsdn-haproxy-52-01`, `ffsdn-haproxy-52-02`.
|
||||||
|
|
||||||
|
## Lab IP assignments (live, validated 2026-02-24)
|
||||||
|
|
||||||
|
| Component | IP |
|
||||||
|
|-----------|-----|
|
||||||
|
| VIP | 192.168.103.200 |
|
||||||
|
| HAProxy 01 | 10.10.10.50 (oc-node-01) |
|
||||||
|
| HAProxy 02 | 10.10.10.51 (oc-node-02) |
|
||||||
|
| nginx-lb-01 | 10.10.10.60 |
|
||||||
|
| nginx-lb-02 | 10.10.10.61 |
|
||||||
|
| nginx-lb-03 | 10.10.10.62 |
|
||||||
|
|
||||||
|
## Validated gotchas
|
||||||
|
|
||||||
|
- **OVN LB not created by Aether**: Aether deploys HAProxy containers and
|
||||||
|
ACLs but does NOT create the OVN load balancer (`incus network load-balancer`).
|
||||||
|
You must create it manually after deployment. See guide Step 3.
|
||||||
|
- **HTTP mode health checks**: Aether generates `option httpchk` (sends
|
||||||
|
`OPTIONS /`). Nginx returns 405 → backends marked DOWN. **Use TCP mode**
|
||||||
|
for nginx backends, or configure nginx to accept OPTIONS.
|
||||||
|
- **HP-01 may not auto-start**: After deploy, `ffsdn-haproxy-52-01` was
|
||||||
|
sometimes found Stopped. Check both containers post-deploy. Not always
|
||||||
|
reproducible — both started on the 2026-02-24 re-validation.
|
||||||
|
- **OVN VIP external access**: Aether ACLs only allow ingress from
|
||||||
|
`source: VIP_IP`. External traffic preserves original client IP → rejected.
|
||||||
|
**Fix**: add broad ingress rule to each HAProxy ACL:
|
||||||
|
`incus network acl rule add <remote>:<acl> ingress action=allow protocol=tcp destination_port=80,443`
|
||||||
|
Must re-apply after infrastructure redeploy.
|
||||||
|
- **Internal VIP not reachable**: The manually-created OVN LB handles
|
||||||
|
external-to-internal traffic but does NOT hairpin for internal-to-VIP
|
||||||
|
requests from the same logical switch. Use HAProxy localhost for internal tests.
|
||||||
|
- **OVN LB has no health checking**: When one HAProxy instance is stopped,
|
||||||
|
the OVN LB continues routing ~50% of connections to the dead backend,
|
||||||
|
causing timeouts. Failover is not instant. Consider Keepalived mode for
|
||||||
|
production deployments requiring true active/standby failover.
|
||||||
|
- **Static IPs on OVN**: Use `ipv4.address` device key on the NIC device,
|
||||||
|
not systemd-networkd: `incus launch ... -d eth0,ipv4.address=10.10.10.60`.
|
||||||
|
|
||||||
|
## Key differences from deploy-awx
|
||||||
|
|
||||||
|
- **No VM creation**: HAProxy uses Aether-managed containers, not a standalone VM.
|
||||||
|
- **Playwright auth**: Uses browser automation, not JWT or basic auth.
|
||||||
|
- **No --heal action**: HAProxy is fully Aether-managed. Recovery = `--cleanup` + `--deploy`.
|
||||||
|
- **Backends via incus CLI**: Test nginx backends created directly with `ipv4.address` device key.
|
||||||
|
|
@ -0,0 +1,8 @@
|
||||||
|
{
|
||||||
|
"mcpServers": {
|
||||||
|
"playwright": {
|
||||||
|
"command": "npx",
|
||||||
|
"args": ["@playwright/mcp@latest"]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
41
CLAUDE.md
41
CLAUDE.md
|
|
@ -26,10 +26,12 @@ incus-contrib/
|
||||||
│ ├── incusos-seed # Seed archive generator (Linux + macOS)
|
│ ├── incusos-seed # Seed archive generator (Linux + macOS)
|
||||||
│ ├── incusos-proxmox # Declarative Proxmox VM deployment + lab lifecycle
|
│ ├── incusos-proxmox # Declarative Proxmox VM deployment + lab lifecycle
|
||||||
│ ├── deploy-awx # AWX deployment + management on Incus cluster
|
│ ├── deploy-awx # AWX deployment + management on Incus cluster
|
||||||
|
│ ├── deploy-haproxy # HAProxy LB deployment + management via Aether
|
||||||
│ ├── awx-manifests/ # K8s manifests for AWX Operator + instance
|
│ ├── awx-manifests/ # K8s manifests for AWX Operator + instance
|
||||||
│ ├── helpers/
|
│ ├── helpers/
|
||||||
│ │ ├── proxmox-screenshot # VMID -> PNG console screenshot
|
│ │ ├── proxmox-screenshot # VMID -> PNG console screenshot
|
||||||
│ │ └── proxmox-api # Authenticated API calls (handles ! in token)
|
│ │ ├── proxmox-api # Authenticated API calls (handles ! in token)
|
||||||
|
│ │ └── aether-browser # Playwright browser automation for Aether web UI
|
||||||
│ ├── lab-test # Guided lab validation (12 test phases)
|
│ ├── lab-test # Guided lab validation (12 test phases)
|
||||||
│ ├── observe-deploy # Single-VM deploy with console screenshots
|
│ ├── observe-deploy # Single-VM deploy with console screenshots
|
||||||
│ ├── proxmox.yaml # Proxmox connection (gitignored)
|
│ ├── proxmox.yaml # Proxmox connection (gitignored)
|
||||||
|
|
@ -95,6 +97,42 @@ curl -sk http://192.168.102.161:30080/api/v2/jobs/{JOB_ID}/stdout/?format=txt \
|
||||||
-H "Authorization: Bearer $AWX_TOKEN"
|
-H "Authorization: Bearer $AWX_TOKEN"
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### Aether browser automation (Playwright)
|
||||||
|
|
||||||
|
Many Aether features (HAProxy management, blueprints, deploys) are **not in
|
||||||
|
the JWT API** — they use session-authenticated routes with CSRF protection
|
||||||
|
that curl cannot handle reliably. **USE PLAYWRIGHT** for these interactions.
|
||||||
|
|
||||||
|
**MCP server**: Configured in `.mcp.json` — provides `browser_navigate`,
|
||||||
|
`browser_click`, `browser_fill`, `browser_screenshot` etc. as tools when
|
||||||
|
the Playwright MCP server is running. Prefer MCP tools when available.
|
||||||
|
|
||||||
|
**Helper script**: `incusos/helpers/aether-browser` provides standalone
|
||||||
|
Playwright automation when MCP tools aren't loaded:
|
||||||
|
```bash
|
||||||
|
source env # loads AETHER_ADMIN_PASSWORD
|
||||||
|
NODE_PATH=~/node_modules node incusos/helpers/aether-browser <action> [args]
|
||||||
|
```
|
||||||
|
|
||||||
|
Actions:
|
||||||
|
- `login` — authenticate and save session cookies to `/tmp/aether-cookies.json`
|
||||||
|
- `screenshot <path>` — navigate to path, take screenshot
|
||||||
|
- `navigate <path>` — navigate and show page title/URL
|
||||||
|
- `haproxy-status` — screenshot HAProxy management page
|
||||||
|
- `haproxy-images` — list built HAProxy images via session API
|
||||||
|
- `api-get <endpoint>` — authenticated GET using browser session
|
||||||
|
- `api-post <endpoint> <json>` — authenticated POST with CSRF handling
|
||||||
|
- `eval <js>` — evaluate JavaScript on the current page
|
||||||
|
|
||||||
|
**When to use what**:
|
||||||
|
- JWT API (`/api/*` endpoints): use `curl` with bearer token
|
||||||
|
- Session-authenticated routes (`/haproxy/*`, UI actions): use Playwright
|
||||||
|
- Never use curl for session auth — CSRF protection requires browser cookies
|
||||||
|
|
||||||
|
**Key technical detail**: Aether login is at `/` (root), NOT `/login`.
|
||||||
|
The form POSTs to `/login`. CSRF token is in a hidden `<input>` field,
|
||||||
|
not in cookies.
|
||||||
|
|
||||||
## Critical safety rules
|
## Critical safety rules
|
||||||
|
|
||||||
- **Proxmox SSH is screenshots-only.** Full rules in `.claude/rules/proxmox-ssh-rules.md`.
|
- **Proxmox SSH is screenshots-only.** Full rules in `.claude/rules/proxmox-ssh-rules.md`.
|
||||||
|
|
@ -146,6 +184,7 @@ which files are being edited:
|
||||||
| `operations-center.md` | OC guide, incusos-proxmox, OC example configs |
|
| `operations-center.md` | OC guide, incusos-proxmox, OC example configs |
|
||||||
| `networking-storage.md` | networking, storage, migration, UTM guides |
|
| `networking-storage.md` | networking, storage, migration, UTM guides |
|
||||||
| `awx-integration.md` | ansible/, deploy-awx, awx-manifests, AWX guide |
|
| `awx-integration.md` | ansible/, deploy-awx, awx-manifests, AWX guide |
|
||||||
|
| `haproxy-lb.md` | deploy-haproxy, HAProxy guide |
|
||||||
| `proxmox-ssh-rules.md` | **Always loaded** (no paths filter) |
|
| `proxmox-ssh-rules.md` | **Always loaded** (no paths filter) |
|
||||||
| `lab-infrastructure.md` | incusos-proxmox, lab-test, examples |
|
| `lab-infrastructure.md` | incusos-proxmox, lab-test, examples |
|
||||||
|
|
||||||
|
|
|
||||||
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,306 @@
|
||||||
|
#!/usr/bin/env node
|
||||||
|
// aether-browser: Playwright helper for Aether web UI automation
|
||||||
|
// Usage: aether-browser <action> [args...]
|
||||||
|
//
|
||||||
|
// Actions:
|
||||||
|
// login - Login and save session cookies
|
||||||
|
// screenshot <path> - Take screenshot of current page
|
||||||
|
// haproxy-status - Check HAProxy infrastructure status
|
||||||
|
// haproxy-images - List HAProxy images
|
||||||
|
// navigate <url> - Navigate to URL and return page content
|
||||||
|
// eval <js> - Evaluate JavaScript on current page
|
||||||
|
//
|
||||||
|
// Environment:
|
||||||
|
// AETHER_URL - Aether base URL (default: https://192.168.102.160:8443)
|
||||||
|
// AETHER_ADMIN_PASSWORD - Admin password (required)
|
||||||
|
// AETHER_COOKIE_FILE - Cookie persistence file (default: /tmp/aether-cookies.json)
|
||||||
|
// HEADLESS - Set to "false" for visible browser (default: true)
|
||||||
|
|
||||||
|
const { chromium } = require('playwright-core');
|
||||||
|
const fs = require('fs');
|
||||||
|
const path = require('path');
|
||||||
|
|
||||||
|
const AETHER_URL = process.env.AETHER_URL || 'https://192.168.102.160:8443';
|
||||||
|
const PASSWORD = process.env.AETHER_ADMIN_PASSWORD;
|
||||||
|
const COOKIE_FILE = process.env.AETHER_COOKIE_FILE || '/tmp/aether-cookies.json';
|
||||||
|
const HEADLESS = process.env.HEADLESS !== 'false';
|
||||||
|
const SCREENSHOT_DIR = '/tmp';
|
||||||
|
|
||||||
|
if (!PASSWORD) {
|
||||||
|
console.error('ERROR: AETHER_ADMIN_PASSWORD not set. Source the env file first.');
|
||||||
|
process.exit(1);
|
||||||
|
}
|
||||||
|
|
||||||
|
async function getBrowser() {
|
||||||
|
// Use Playwright's bundled Chromium
|
||||||
|
const browserPath = require('playwright-core').chromium.executablePath();
|
||||||
|
return await chromium.launch({
|
||||||
|
headless: HEADLESS,
|
||||||
|
executablePath: browserPath,
|
||||||
|
args: ['--ignore-certificate-errors', '--no-sandbox'],
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
async function getContextWithCookies(browser) {
|
||||||
|
const context = await browser.newContext({
|
||||||
|
ignoreHTTPSErrors: true,
|
||||||
|
viewport: { width: 1280, height: 800 },
|
||||||
|
});
|
||||||
|
|
||||||
|
// Restore cookies if available
|
||||||
|
if (fs.existsSync(COOKIE_FILE)) {
|
||||||
|
try {
|
||||||
|
const cookies = JSON.parse(fs.readFileSync(COOKIE_FILE, 'utf8'));
|
||||||
|
await context.addCookies(cookies);
|
||||||
|
} catch (e) {
|
||||||
|
// Stale cookies, will re-login
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return context;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function saveCookies(context) {
|
||||||
|
const cookies = await context.cookies();
|
||||||
|
fs.writeFileSync(COOKIE_FILE, JSON.stringify(cookies, null, 2));
|
||||||
|
}
|
||||||
|
|
||||||
|
async function login(page) {
|
||||||
|
console.log(`Navigating to ${AETHER_URL}/...`);
|
||||||
|
await page.goto(`${AETHER_URL}/`, { waitUntil: 'networkidle' });
|
||||||
|
|
||||||
|
// Check if already logged in
|
||||||
|
const url = page.url();
|
||||||
|
if (!url.includes('login') && !await page.locator('input[name="csrf_token"]').count()) {
|
||||||
|
console.log('Already logged in (session cookies valid).');
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log('Login page detected. Filling credentials...');
|
||||||
|
|
||||||
|
// Fill the login form
|
||||||
|
await page.fill('input[name="username"]', 'admin');
|
||||||
|
await page.fill('input[name="password"]', PASSWORD);
|
||||||
|
|
||||||
|
// Submit
|
||||||
|
await page.click('button[type="submit"], input[type="submit"]');
|
||||||
|
|
||||||
|
// Wait for navigation after login
|
||||||
|
await page.waitForLoadState('networkidle');
|
||||||
|
|
||||||
|
// Check success
|
||||||
|
const afterUrl = page.url();
|
||||||
|
if (afterUrl.includes('login')) {
|
||||||
|
console.error('ERROR: Login failed - still on login page.');
|
||||||
|
await page.screenshot({ path: `${SCREENSHOT_DIR}/aether-login-fail.png` });
|
||||||
|
console.error(`Screenshot saved to ${SCREENSHOT_DIR}/aether-login-fail.png`);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log(`Login successful. Redirected to: ${afterUrl}`);
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function doAction(action, args) {
|
||||||
|
const browser = await getBrowser();
|
||||||
|
const context = await getContextWithCookies(browser);
|
||||||
|
const page = await context.newPage();
|
||||||
|
|
||||||
|
try {
|
||||||
|
switch (action) {
|
||||||
|
case 'login': {
|
||||||
|
const success = await login(page);
|
||||||
|
if (success) {
|
||||||
|
await saveCookies(context);
|
||||||
|
console.log(`Session cookies saved to ${COOKIE_FILE}`);
|
||||||
|
}
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
|
||||||
|
case 'screenshot': {
|
||||||
|
const loginOk = await login(page);
|
||||||
|
if (!loginOk) break;
|
||||||
|
await saveCookies(context);
|
||||||
|
|
||||||
|
const targetUrl = args[0] || '/';
|
||||||
|
if (targetUrl.startsWith('/')) {
|
||||||
|
await page.goto(`${AETHER_URL}${targetUrl}`, { waitUntil: 'networkidle' });
|
||||||
|
}
|
||||||
|
|
||||||
|
const outPath = args[1] || `${SCREENSHOT_DIR}/aether-screenshot.png`;
|
||||||
|
await page.screenshot({ path: outPath, fullPage: true });
|
||||||
|
console.log(`Screenshot saved to ${outPath}`);
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
|
||||||
|
case 'haproxy-status': {
|
||||||
|
const loginOk = await login(page);
|
||||||
|
if (!loginOk) break;
|
||||||
|
await saveCookies(context);
|
||||||
|
|
||||||
|
// Navigate to HAProxy infrastructure page
|
||||||
|
console.log('Navigating to HAProxy management...');
|
||||||
|
await page.goto(`${AETHER_URL}/haproxy`, { waitUntil: 'networkidle' });
|
||||||
|
await page.screenshot({ path: `${SCREENSHOT_DIR}/aether-haproxy.png`, fullPage: true });
|
||||||
|
console.log(`Screenshot: ${SCREENSHOT_DIR}/aether-haproxy.png`);
|
||||||
|
|
||||||
|
// Try to extract status info from the page
|
||||||
|
const content = await page.textContent('body');
|
||||||
|
// Print relevant sections
|
||||||
|
const lines = content.split('\n').filter(l => l.trim());
|
||||||
|
console.log('\n--- Page content (filtered) ---');
|
||||||
|
for (const line of lines.slice(0, 50)) {
|
||||||
|
console.log(line.trim());
|
||||||
|
}
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
|
||||||
|
case 'haproxy-images': {
|
||||||
|
const loginOk = await login(page);
|
||||||
|
if (!loginOk) break;
|
||||||
|
await saveCookies(context);
|
||||||
|
|
||||||
|
console.log('Fetching HAProxy images...');
|
||||||
|
// Use the session to make an API call
|
||||||
|
const response = await page.evaluate(async () => {
|
||||||
|
const resp = await fetch('/haproxy/images', {
|
||||||
|
credentials: 'same-origin',
|
||||||
|
});
|
||||||
|
return { status: resp.status, body: await resp.text() };
|
||||||
|
});
|
||||||
|
console.log(`Status: ${response.status}`);
|
||||||
|
console.log(response.body);
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
|
||||||
|
case 'navigate': {
|
||||||
|
const loginOk = await login(page);
|
||||||
|
if (!loginOk) break;
|
||||||
|
await saveCookies(context);
|
||||||
|
|
||||||
|
const navUrl = args[0] || '/';
|
||||||
|
const fullUrl = navUrl.startsWith('http') ? navUrl : `${AETHER_URL}${navUrl}`;
|
||||||
|
console.log(`Navigating to ${fullUrl}...`);
|
||||||
|
await page.goto(fullUrl, { waitUntil: 'networkidle' });
|
||||||
|
|
||||||
|
const outPath = `${SCREENSHOT_DIR}/aether-navigate.png`;
|
||||||
|
await page.screenshot({ path: outPath, fullPage: true });
|
||||||
|
console.log(`Screenshot: ${outPath}`);
|
||||||
|
|
||||||
|
const title = await page.title();
|
||||||
|
console.log(`Title: ${title}`);
|
||||||
|
console.log(`URL: ${page.url()}`);
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
|
||||||
|
case 'api-get': {
|
||||||
|
const loginOk = await login(page);
|
||||||
|
if (!loginOk) break;
|
||||||
|
await saveCookies(context);
|
||||||
|
|
||||||
|
const endpoint = args[0];
|
||||||
|
if (!endpoint) {
|
||||||
|
console.error('Usage: aether-browser api-get <endpoint>');
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
|
||||||
|
const response = await page.evaluate(async (url) => {
|
||||||
|
const resp = await fetch(url, { credentials: 'same-origin' });
|
||||||
|
const text = await resp.text();
|
||||||
|
return { status: resp.status, contentType: resp.headers.get('content-type'), body: text };
|
||||||
|
}, endpoint);
|
||||||
|
|
||||||
|
console.log(`GET ${endpoint} -> ${response.status}`);
|
||||||
|
if (response.contentType && response.contentType.includes('json')) {
|
||||||
|
try {
|
||||||
|
console.log(JSON.stringify(JSON.parse(response.body), null, 2));
|
||||||
|
} catch {
|
||||||
|
console.log(response.body);
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
console.log(response.body);
|
||||||
|
}
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
|
||||||
|
case 'api-post': {
|
||||||
|
const loginOk = await login(page);
|
||||||
|
if (!loginOk) break;
|
||||||
|
await saveCookies(context);
|
||||||
|
|
||||||
|
const endpoint = args[0];
|
||||||
|
const bodyJson = args[1];
|
||||||
|
if (!endpoint || !bodyJson) {
|
||||||
|
console.error('Usage: aether-browser api-post <endpoint> <json-body>');
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Get CSRF token from meta tag or cookie
|
||||||
|
const csrfToken = await page.evaluate(() => {
|
||||||
|
const meta = document.querySelector('meta[name="csrf-token"]');
|
||||||
|
if (meta) return meta.getAttribute('content');
|
||||||
|
// Try from cookie
|
||||||
|
const match = document.cookie.match(/csrf_token=([^;]+)/);
|
||||||
|
return match ? match[1] : '';
|
||||||
|
});
|
||||||
|
|
||||||
|
const response = await page.evaluate(async ({ url, body, csrf }) => {
|
||||||
|
const headers = {
|
||||||
|
'Content-Type': 'application/json',
|
||||||
|
};
|
||||||
|
if (csrf) {
|
||||||
|
headers['X-CSRFToken'] = csrf;
|
||||||
|
headers['X-CSRF-Token'] = csrf;
|
||||||
|
}
|
||||||
|
const resp = await fetch(url, {
|
||||||
|
method: 'POST',
|
||||||
|
credentials: 'same-origin',
|
||||||
|
headers,
|
||||||
|
body,
|
||||||
|
});
|
||||||
|
const text = await resp.text();
|
||||||
|
return { status: resp.status, body: text };
|
||||||
|
}, { url: endpoint, body: bodyJson, csrf: csrfToken });
|
||||||
|
|
||||||
|
console.log(`POST ${endpoint} -> ${response.status}`);
|
||||||
|
try {
|
||||||
|
console.log(JSON.stringify(JSON.parse(response.body), null, 2));
|
||||||
|
} catch {
|
||||||
|
console.log(response.body);
|
||||||
|
}
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
|
||||||
|
case 'eval': {
|
||||||
|
const loginOk = await login(page);
|
||||||
|
if (!loginOk) break;
|
||||||
|
await saveCookies(context);
|
||||||
|
|
||||||
|
const js = args.join(' ');
|
||||||
|
const result = await page.evaluate(js);
|
||||||
|
console.log(JSON.stringify(result, null, 2));
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
|
||||||
|
default:
|
||||||
|
console.error(`Unknown action: ${action}`);
|
||||||
|
console.error('Actions: login, screenshot, haproxy-status, haproxy-images, navigate, api-get, api-post, eval');
|
||||||
|
process.exit(1);
|
||||||
|
}
|
||||||
|
} finally {
|
||||||
|
await browser.close();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Main
|
||||||
|
const [action, ...args] = process.argv.slice(2);
|
||||||
|
if (!action) {
|
||||||
|
console.error('Usage: aether-browser <action> [args...]');
|
||||||
|
console.error('Actions: login, screenshot, haproxy-status, haproxy-images, navigate, api-get, api-post, eval');
|
||||||
|
process.exit(1);
|
||||||
|
}
|
||||||
|
|
||||||
|
doAction(action, args).catch(err => {
|
||||||
|
console.error('Fatal error:', err.message);
|
||||||
|
process.exit(1);
|
||||||
|
});
|
||||||
|
|
@ -0,0 +1,780 @@
|
||||||
|
# HAProxy Load Balancing — Aether-Managed HA Proxy on OVN
|
||||||
|
|
||||||
|
Aether includes a full HAProxy management system that deploys a highly
|
||||||
|
available load balancer pair on OVN networks. It handles image builds,
|
||||||
|
HA pair deployment, VIP management, L7 service configuration, and health
|
||||||
|
monitoring — all through its web UI and session-authenticated API.
|
||||||
|
|
||||||
|
This guide covers deploying HAProxy infrastructure on the lab cluster,
|
||||||
|
creating test backends, configuring services, and verifying load balancing.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
Client / LAN traffic
|
||||||
|
|
|
||||||
|
v
|
||||||
|
+---------------------------+
|
||||||
|
| UPLINK VIP Address |
|
||||||
|
| 192.168.103.200 |
|
||||||
|
+---------------------------+
|
||||||
|
|
|
||||||
|
v
|
||||||
|
+---------------------------+
|
||||||
|
| OVN Load Balancer |
|
||||||
|
| (distributes to HA |
|
||||||
|
| pair for failover) |
|
||||||
|
+---------------------------+
|
||||||
|
/ \
|
||||||
|
/ \
|
||||||
|
v v
|
||||||
|
+-----------------+ +-----------------+
|
||||||
|
| HAProxy 01 | | HAProxy 02 |
|
||||||
|
| 10.10.10.50 | | 10.10.10.51 |
|
||||||
|
| (container) | | (container) |
|
||||||
|
+-----------------+ +-----------------+
|
||||||
|
\ | /
|
||||||
|
\ | /
|
||||||
|
v v v
|
||||||
|
+----------+ +----------+ +----------+
|
||||||
|
| nginx-01 | | nginx-02 | | nginx-03 |
|
||||||
|
| .60 :80 | | .61 :80 | | .62 :80 |
|
||||||
|
+----------+ +----------+ +----------+
|
||||||
|
(backend servers on net-prod)
|
||||||
|
```
|
||||||
|
|
||||||
|
### OVN native LB vs HAProxy
|
||||||
|
|
||||||
|
| Feature | OVN native LB | Aether HAProxy |
|
||||||
|
|---------|---------------|----------------|
|
||||||
|
| Layer | L4 (TCP/UDP) | L4 + L7 (HTTP/HTTPS) |
|
||||||
|
| Health checks | None | HTTP, TCP, configurable interval |
|
||||||
|
| SSL termination | No | Yes (HTTPS Termination mode) |
|
||||||
|
| Sticky sessions | No | Yes (cookie-based) |
|
||||||
|
| Load balancing | Connection hash | Round robin, least conn, source hash, first available |
|
||||||
|
| SNI routing | No | Yes (multiple domains on same VIP:port) |
|
||||||
|
| Compression | No | Yes (gzip for text content) |
|
||||||
|
| Rate limiting | No | Yes (per-IP) |
|
||||||
|
| Management | `incus network load-balancer` CLI | Aether UI + session API |
|
||||||
|
| HA | Single OVN LB (distributed) | HA pair with OVN LB failover |
|
||||||
|
|
||||||
|
Use OVN native LB for simple L4 distribution. Use HAProxy when you need
|
||||||
|
health checks, L7 features, SSL, sticky sessions, or advanced traffic management.
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
- Incus cluster registered in Aether with OVN networking configured
|
||||||
|
- OVN network with a physical UPLINK (for VIP addresses)
|
||||||
|
- Available IP addresses on the OVN network for HAProxy instances
|
||||||
|
- Available VIP address(es) from the UPLINK range
|
||||||
|
- Internet access from the cluster (for HAProxy image builds)
|
||||||
|
- `incus` CLI configured with cluster remote
|
||||||
|
- Aether admin access (Full Admin or LB Admin role)
|
||||||
|
|
||||||
|
### Resource requirements
|
||||||
|
|
||||||
|
| Resource | Per HAProxy instance | Notes |
|
||||||
|
|----------|---------------------|-------|
|
||||||
|
| CPU | 2 cores (default) | HAProxy is very CPU-efficient |
|
||||||
|
| Memory | 1 GiB (default) | Increase for many concurrent connections |
|
||||||
|
| Disk | Minimal | Container image, no persistent storage needed |
|
||||||
|
|
||||||
|
## Lab details
|
||||||
|
|
||||||
|
| Setting | Value |
|
||||||
|
|---------|-------|
|
||||||
|
| Cluster | oc-lab-cluster (ID: 52) |
|
||||||
|
| OVN network | net-prod (10.10.10.0/24) |
|
||||||
|
| UPLINK | UPLINK (physical) |
|
||||||
|
| VIP range | 192.168.103.200-210 |
|
||||||
|
| HAProxy 01 IP | 10.10.10.50 |
|
||||||
|
| HAProxy 02 IP | 10.10.10.51 |
|
||||||
|
| Backend IPs | 10.10.10.60, 10.10.10.61, 10.10.10.62 |
|
||||||
|
| HAProxy containers | ffsdn-haproxy-52-01, ffsdn-haproxy-52-02 |
|
||||||
|
| Aether URL | https://192.168.102.160:8443 |
|
||||||
|
| Cluster remote | oc-node-01 |
|
||||||
|
|
||||||
|
## Automated path
|
||||||
|
|
||||||
|
The `incusos/deploy-haproxy` script automates the entire deployment:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check prerequisites
|
||||||
|
deploy-haproxy --doctor
|
||||||
|
|
||||||
|
# Preview deployment
|
||||||
|
deploy-haproxy --deploy --dry-run
|
||||||
|
|
||||||
|
# Full deployment (backends + image + infrastructure + service)
|
||||||
|
deploy-haproxy --deploy
|
||||||
|
|
||||||
|
# Check health
|
||||||
|
deploy-haproxy --status
|
||||||
|
|
||||||
|
# Clean removal
|
||||||
|
deploy-haproxy --cleanup
|
||||||
|
```
|
||||||
|
|
||||||
|
The manual steps below are for reference, troubleshooting, and understanding
|
||||||
|
what the script does.
|
||||||
|
|
||||||
|
## Step 1: Deploy test backends
|
||||||
|
|
||||||
|
Create 3 nginx containers on net-prod to serve as load-balanced backends.
|
||||||
|
Each gets a unique index page so we can verify traffic distribution.
|
||||||
|
|
||||||
|
**Important**: The backend IPs (10.10.10.60-62) must not already be in use
|
||||||
|
on the OVN network. Check the Aether UI (HAProxy > select cluster) for
|
||||||
|
used IPs before deploying, or use `deploy-haproxy --doctor` which checks
|
||||||
|
for you.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
REMOTE="oc-node-01"
|
||||||
|
|
||||||
|
for i in 60 61 62; do
|
||||||
|
NAME="nginx-lb-$(printf '%02d' $((i - 59)))"
|
||||||
|
IP="10.10.10.${i}"
|
||||||
|
|
||||||
|
# Launch container on OVN network with static IP via device key
|
||||||
|
incus launch images:debian/12 "${REMOTE}:${NAME}" \
|
||||||
|
--network net-prod \
|
||||||
|
--storage local \
|
||||||
|
-d root,size=2GiB \
|
||||||
|
-d eth0,ipv4.address="${IP}" \
|
||||||
|
-c limits.cpu=1 \
|
||||||
|
-c limits.memory=256MiB
|
||||||
|
|
||||||
|
# Wait for container agent
|
||||||
|
echo "Waiting for ${NAME}..."
|
||||||
|
while ! incus exec "${REMOTE}:${NAME}" -- true 2>/dev/null; do
|
||||||
|
sleep 2
|
||||||
|
done
|
||||||
|
|
||||||
|
# Install nginx + curl, create unique index page
|
||||||
|
incus exec "${REMOTE}:${NAME}" -- bash -c "
|
||||||
|
export DEBIAN_FRONTEND=noninteractive
|
||||||
|
apt-get update -qq
|
||||||
|
apt-get install -y -qq nginx curl 2>&1 | tail -1
|
||||||
|
|
||||||
|
cat > /var/www/html/index.html << HTML
|
||||||
|
<html><body>
|
||||||
|
<h1>Backend: ${NAME}</h1>
|
||||||
|
<p>IP: ${IP}</p>
|
||||||
|
</body></html>
|
||||||
|
HTML
|
||||||
|
|
||||||
|
systemctl enable nginx
|
||||||
|
systemctl start nginx
|
||||||
|
"
|
||||||
|
|
||||||
|
echo "Backend ${NAME} deployed at ${IP}"
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
Verify all backends respond (curling from one backend to the others
|
||||||
|
confirms OVN cross-container connectivity):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Wait for static IPs to take effect
|
||||||
|
sleep 5
|
||||||
|
|
||||||
|
for i in 60 61 62; do
|
||||||
|
incus exec "${REMOTE}:nginx-lb-01" -- \
|
||||||
|
curl -s --connect-timeout 3 "http://10.10.10.${i}/" \
|
||||||
|
| grep -o 'Backend: [^<]*'
|
||||||
|
done
|
||||||
|
# Expected:
|
||||||
|
# Backend: nginx-lb-01
|
||||||
|
# Backend: nginx-lb-02
|
||||||
|
# Backend: nginx-lb-03
|
||||||
|
```
|
||||||
|
|
||||||
|
If curl fails, the static IP may not have taken effect yet. Verify with
|
||||||
|
`incus exec "${REMOTE}:nginx-lb-01" -- ip addr show eth0`.
|
||||||
|
|
||||||
|
## Step 2: Build HAProxy image
|
||||||
|
|
||||||
|
Aether builds a base HAProxy container image and stores it internally.
|
||||||
|
The image must be pushed to each target cluster before infrastructure
|
||||||
|
can be deployed.
|
||||||
|
|
||||||
|
**Note**: The HAProxy management features are not in Aether's JWT API
|
||||||
|
(`/api/swagger.yaml`). They use session-authenticated routes discovered
|
||||||
|
from the UI JavaScript. The script uses session auth automatically.
|
||||||
|
|
||||||
|
### Via Aether UI
|
||||||
|
|
||||||
|
1. Navigate to **HAProxy** in the main menu
|
||||||
|
2. In **HAProxy Base Images**, click **Build New Image**
|
||||||
|
3. Select **Build on Cluster**: `oc-lab-cluster`
|
||||||
|
4. Enter **Version**: `1.0.0`
|
||||||
|
5. Click **Build Image**
|
||||||
|
6. Wait 5-10 minutes for the build to complete
|
||||||
|
7. Click **Push to Cluster** and select `oc-lab-cluster`
|
||||||
|
8. Click **Set Current** on the new image
|
||||||
|
|
||||||
|
### Via session API
|
||||||
|
|
||||||
|
The build process uses session cookies + CSRF token authentication:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
AETHER_URL="https://192.168.102.160:8443"
|
||||||
|
COOKIE_JAR="/tmp/aether-cookies.txt"
|
||||||
|
|
||||||
|
# Step 1: Get login page and extract CSRF token from hidden form field
|
||||||
|
# Note: the login form is at "/" (root), NOT "/login". GET /login returns 404.
|
||||||
|
CSRF=$(curl -sSk -c "$COOKIE_JAR" "${AETHER_URL}/" \
|
||||||
|
| grep -o 'name="csrf_token" value="[^"]*"' \
|
||||||
|
| sed 's/.*value="//;s/"//')
|
||||||
|
|
||||||
|
# Step 2: Login with CSRF token (form-urlencoded, NOT JSON)
|
||||||
|
curl -sSk -b "$COOKIE_JAR" -c "$COOKIE_JAR" \
|
||||||
|
-X POST "${AETHER_URL}/login" \
|
||||||
|
-d "csrf_token=${CSRF}&username=admin&password=${AETHER_ADMIN_PASSWORD}" \
|
||||||
|
-L -o /dev/null
|
||||||
|
|
||||||
|
# Step 3: Get fresh CSRF token for API calls
|
||||||
|
CSRF=$(curl -sSk -b "$COOKIE_JAR" "${AETHER_URL}/haproxy" \
|
||||||
|
| grep -o 'name="csrf_token" value="[^"]*"' \
|
||||||
|
| sed 's/.*value="//;s/"//')
|
||||||
|
|
||||||
|
# Step 4: Build image
|
||||||
|
curl -sSk -b "$COOKIE_JAR" \
|
||||||
|
-H "X-CSRF-Token: ${CSRF}" \
|
||||||
|
-H "Referer: ${AETHER_URL}/haproxy" \
|
||||||
|
-X POST "${AETHER_URL}/haproxy/image/build" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"cluster_id": 52, "version": "1.0.0"}'
|
||||||
|
|
||||||
|
# Step 5: Poll build status until complete
|
||||||
|
while true; do
|
||||||
|
STATUS=$(curl -sSk -b "$COOKIE_JAR" \
|
||||||
|
-H "X-CSRF-Token: ${CSRF}" \
|
||||||
|
"${AETHER_URL}/haproxy/image/build-status" \
|
||||||
|
| python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('status','unknown'))")
|
||||||
|
echo "Build status: $STATUS"
|
||||||
|
[[ "$STATUS" == "completed" || "$STATUS" == "failed" ]] && break
|
||||||
|
sleep 10
|
||||||
|
done
|
||||||
|
|
||||||
|
# Step 6: Get image ID and push to cluster
|
||||||
|
IMAGE_ID=$(curl -sSk -b "$COOKIE_JAR" \
|
||||||
|
-H "X-CSRF-Token: ${CSRF}" \
|
||||||
|
"${AETHER_URL}/haproxy/images" \
|
||||||
|
| python3 -c "import sys,json; imgs=json.load(sys.stdin); print(imgs[-1]['id'] if imgs else '')")
|
||||||
|
|
||||||
|
curl -sSk -b "$COOKIE_JAR" \
|
||||||
|
-H "X-CSRF-Token: ${CSRF}" \
|
||||||
|
-H "Referer: ${AETHER_URL}/haproxy" \
|
||||||
|
-X POST "${AETHER_URL}/haproxy/image/push" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d "{\"image_id\": ${IMAGE_ID}, \"cluster_id\": 52}"
|
||||||
|
|
||||||
|
# Step 7: Set as current version
|
||||||
|
curl -sSk -b "$COOKIE_JAR" \
|
||||||
|
-H "X-CSRF-Token: ${CSRF}" \
|
||||||
|
-H "Referer: ${AETHER_URL}/haproxy" \
|
||||||
|
-X POST "${AETHER_URL}/haproxy/image/set-current" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d "{\"image_id\": ${IMAGE_ID}}"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Critical**: The CSRF token comes from a hidden HTML form field
|
||||||
|
(`<input name="csrf_token" value="...">`), NOT from the cookie value.
|
||||||
|
The cookie contains the session ID; the form field contains the CSRF token.
|
||||||
|
|
||||||
|
## Step 3: Deploy HAProxy infrastructure
|
||||||
|
|
||||||
|
Deploy the HA pair on the cluster, then create the OVN load balancer.
|
||||||
|
|
||||||
|
### Via Aether UI
|
||||||
|
|
||||||
|
1. Navigate to **HAProxy** > select **oc-lab-cluster** from dropdown
|
||||||
|
2. Click **Deploy HAProxy Infrastructure**
|
||||||
|
3. Fill in:
|
||||||
|
- **OVN Network**: `net-prod`
|
||||||
|
- **Load Balancer VIP**: `192.168.103.200` (from UPLINK range dropdown)
|
||||||
|
- **HAProxy 01 IP**: `10.10.10.50`
|
||||||
|
- **HAProxy 02 IP**: `10.10.10.51`
|
||||||
|
- **CPU Limit**: `2`
|
||||||
|
- **Memory Limit**: `1GB`
|
||||||
|
4. Click **Deploy**
|
||||||
|
|
||||||
|
### Via session API
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -sSk -b "$COOKIE_JAR" \
|
||||||
|
-H "X-CSRF-Token: ${CSRF}" \
|
||||||
|
-H "Referer: ${AETHER_URL}/haproxy" \
|
||||||
|
-X POST "${AETHER_URL}/haproxy/infrastructure/deploy" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{
|
||||||
|
"cluster_id": 52,
|
||||||
|
"ovn_network": "net-prod",
|
||||||
|
"lb_vip": "192.168.103.200",
|
||||||
|
"haproxy_01_ip": "10.10.10.50",
|
||||||
|
"haproxy_02_ip": "10.10.10.51",
|
||||||
|
"cpu_limit": "2",
|
||||||
|
"memory_limit": "1GB"
|
||||||
|
}'
|
||||||
|
```
|
||||||
|
|
||||||
|
This creates:
|
||||||
|
- Two containers: `ffsdn-haproxy-52-01` and `ffsdn-haproxy-52-02`
|
||||||
|
- Per-instance ACL rules allowing traffic from the VIP
|
||||||
|
- A default VIP entry at 192.168.103.200
|
||||||
|
|
||||||
|
**Important**: Aether does NOT create the OVN load balancer automatically.
|
||||||
|
You must create it manually after infrastructure deployment:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
REMOTE="oc-node-01"
|
||||||
|
|
||||||
|
# Create OVN load balancer for the VIP
|
||||||
|
incus network load-balancer create "${REMOTE}:net-prod" 192.168.103.200 \
|
||||||
|
--description "FFSDN HAProxy Load Balancer"
|
||||||
|
|
||||||
|
# Add both HAProxy instances as backends
|
||||||
|
incus network load-balancer backend add "${REMOTE}:net-prod" 192.168.103.200 \
|
||||||
|
haproxy-01 10.10.10.50 80
|
||||||
|
incus network load-balancer backend add "${REMOTE}:net-prod" 192.168.103.200 \
|
||||||
|
haproxy-02 10.10.10.51 80
|
||||||
|
|
||||||
|
# Map port 80 to both backends
|
||||||
|
incus network load-balancer port add "${REMOTE}:net-prod" 192.168.103.200 \
|
||||||
|
tcp 80 haproxy-01,haproxy-02
|
||||||
|
|
||||||
|
# Verify
|
||||||
|
incus network load-balancer list "${REMOTE}:net-prod"
|
||||||
|
```
|
||||||
|
|
||||||
|
Verify infrastructure:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check via API
|
||||||
|
curl -sSk -b "$COOKIE_JAR" \
|
||||||
|
-H "X-CSRF-Token: ${CSRF}" \
|
||||||
|
"${AETHER_URL}/haproxy/infrastructure/52"
|
||||||
|
|
||||||
|
# Check containers exist on cluster
|
||||||
|
incus list oc-node-01: --format csv -c n | grep ffsdn-haproxy
|
||||||
|
# ffsdn-haproxy-52-01
|
||||||
|
# ffsdn-haproxy-52-02
|
||||||
|
```
|
||||||
|
|
||||||
|
### Fix ACL rules for external access
|
||||||
|
|
||||||
|
Aether auto-generates per-instance ACLs that only allow ingress from
|
||||||
|
`source: 192.168.103.200` (the VIP address). This works for traffic
|
||||||
|
originating inside the OVN network — OVN hairpin-SNATs the source to
|
||||||
|
the router IP (.200) when load-balancing back into the same network.
|
||||||
|
However, external traffic (from outside OVN) preserves the original
|
||||||
|
client IP, which doesn't match the ACL and gets **rejected**.
|
||||||
|
|
||||||
|
Add a broader ingress rule to each HAProxy container's ACL:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
REMOTE="oc-node-01"
|
||||||
|
CLUSTER_ID=52
|
||||||
|
|
||||||
|
for n in 01 02; do
|
||||||
|
ACL_NAME="${CLUSTER_ID}-ffsdn-haproxy-${CLUSTER_ID}-${n}-aether-acl"
|
||||||
|
incus network acl rule add "${REMOTE}:${ACL_NAME}" ingress \
|
||||||
|
action=allow protocol=tcp destination_port=80,443 \
|
||||||
|
description="Allow external clients to reach HAProxy"
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
**Why is this needed?** OVN load balancers perform DNAT (rewriting the
|
||||||
|
destination IP from the VIP to a backend), but for external-to-internal
|
||||||
|
traffic they do NOT SNAT the source. The packet arrives at the HAProxy
|
||||||
|
container with the external client's IP as source, which Aether's narrow
|
||||||
|
ACL rule rejects. For internal traffic, OVN *does* hairpin-SNAT the source
|
||||||
|
to .200 (to prevent asymmetric routing within the same logical switch),
|
||||||
|
which is why it works from inside OVN without this fix.
|
||||||
|
|
||||||
|
**Note**: This ACL modification survives container restarts and
|
||||||
|
reconfigurations. However, if you delete and redeploy the HAProxy
|
||||||
|
infrastructure, Aether recreates the ACLs from scratch, so you'll
|
||||||
|
need to re-apply this fix. Check future Aether versions — this may
|
||||||
|
be fixed upstream.
|
||||||
|
|
||||||
|
## Step 4: Create service
|
||||||
|
|
||||||
|
Configure HAProxy to load-balance traffic across the nginx backends.
|
||||||
|
|
||||||
|
**Important: TCP vs HTTP mode** — For nginx backends, use **TCP mode**.
|
||||||
|
HTTP mode generates `option httpchk` which sends `OPTIONS /` requests.
|
||||||
|
Nginx returns `405 Not Allowed` for OPTIONS, causing HAProxy to mark
|
||||||
|
all backends as DOWN. TCP mode uses simple TCP health checks (L4OK).
|
||||||
|
|
||||||
|
### Via Aether UI
|
||||||
|
|
||||||
|
1. In the **Services** section, click **+ Add Service**
|
||||||
|
2. Fill in:
|
||||||
|
- **Service Name**: `web-http`
|
||||||
|
- **Description**: `Load balancer for nginx test backends`
|
||||||
|
- **VIP**: `192.168.103.200 - Default`
|
||||||
|
- **Listen Port**: `80`
|
||||||
|
- **Mode**: `TCP (Layer 4)`
|
||||||
|
- **Balance Method**: `Round Robin`
|
||||||
|
- **Health Check**: enabled
|
||||||
|
3. Add backend servers:
|
||||||
|
- `nginx-lb-01` / `10.10.10.60` / port `80` / weight `100`
|
||||||
|
- `nginx-lb-02` / `10.10.10.61` / port `80` / weight `100`
|
||||||
|
- `nginx-lb-03` / `10.10.10.62` / port `80` / weight `100`
|
||||||
|
4. Click **Create Service**
|
||||||
|
|
||||||
|
### Via session API
|
||||||
|
|
||||||
|
**Note**: The `vip_id` is the numeric VIP ID from the infrastructure
|
||||||
|
deployment (visible in the API response or the Aether UI). For the
|
||||||
|
default VIP created during deployment, retrieve it from
|
||||||
|
`GET /haproxy/infrastructure/52` → `vips[0].id`.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -sSk -b "$COOKIE_JAR" \
|
||||||
|
-H "X-CSRF-Token: ${CSRF}" \
|
||||||
|
-H "Referer: ${AETHER_URL}/haproxy" \
|
||||||
|
-X POST "${AETHER_URL}/haproxy/service" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{
|
||||||
|
"vip_id": 42,
|
||||||
|
"name": "web-http",
|
||||||
|
"description": "HTTP load balancer for nginx test backends",
|
||||||
|
"hostname": "",
|
||||||
|
"listen_port": 80,
|
||||||
|
"mode": "tcp",
|
||||||
|
"balance_method": "roundrobin",
|
||||||
|
"health_check_enabled": true,
|
||||||
|
"session_persistence": false,
|
||||||
|
"backends": [
|
||||||
|
{"name": "nginx-lb-01", "target_ip": "10.10.10.60", "target_port": 80, "weight": 100},
|
||||||
|
{"name": "nginx-lb-02", "target_ip": "10.10.10.61", "target_port": 80, "weight": 100},
|
||||||
|
{"name": "nginx-lb-03", "target_ip": "10.10.10.62", "target_port": 80, "weight": 100}
|
||||||
|
]
|
||||||
|
}'
|
||||||
|
```
|
||||||
|
|
||||||
|
**Note**: After creating or editing a service, Aether may need to reload
|
||||||
|
the HAProxy configuration on both instances. If the service doesn't respond
|
||||||
|
immediately, click **Reload Config** in the Aether UI, or wait a few seconds
|
||||||
|
for automatic propagation.
|
||||||
|
|
||||||
|
## Step 5: Test load balancing
|
||||||
|
|
||||||
|
### Internal connectivity (via HAProxy localhost)
|
||||||
|
|
||||||
|
Test HAProxy directly by curling localhost on each container. This
|
||||||
|
bypasses the OVN load balancer and tests HAProxy's backend routing:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Direct test on HAProxy 01 (bypasses OVN LB)
|
||||||
|
for i in $(seq 1 6); do
|
||||||
|
incus exec oc-node-01:ffsdn-haproxy-52-01 -- \
|
||||||
|
curl -s http://localhost/ | grep -o 'Backend: [^<]*'
|
||||||
|
done
|
||||||
|
# Expected: nginx-lb-01, nginx-lb-02, nginx-lb-03 in rotation
|
||||||
|
|
||||||
|
# Direct test on HAProxy 02
|
||||||
|
incus exec oc-node-01:ffsdn-haproxy-52-02 -- curl -s http://localhost/
|
||||||
|
# Expected: response from one of the backends
|
||||||
|
```
|
||||||
|
|
||||||
|
**Note**: Curling the VIP (192.168.103.200) from within the OVN network
|
||||||
|
does NOT work reliably. The manually-created OVN load balancer handles
|
||||||
|
external-to-internal traffic but does not hairpin for internal-to-VIP
|
||||||
|
requests from the same logical switch. Use localhost for internal tests.
|
||||||
|
|
||||||
|
### External connectivity (from outside OVN)
|
||||||
|
|
||||||
|
After applying the ACL fix in Step 3, the VIP is reachable from any
|
||||||
|
machine on the LAN:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Single request from external machine
|
||||||
|
curl -s http://192.168.103.200/
|
||||||
|
# Expected: response from one of the backends
|
||||||
|
|
||||||
|
# Verify round-robin across all backends
|
||||||
|
for i in $(seq 1 6); do
|
||||||
|
curl -s http://192.168.103.200/ | grep -o '<h1>[^<]*</h1>'
|
||||||
|
done
|
||||||
|
# Expected: all three backends appear in rotation
|
||||||
|
```
|
||||||
|
|
||||||
|
If external access fails with "Connection refused", the ACL fix was
|
||||||
|
not applied. See Step 3 "Fix ACL rules for external access".
|
||||||
|
|
||||||
|
### Backend failover
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Stop one backend
|
||||||
|
incus stop oc-node-01:nginx-lb-03
|
||||||
|
|
||||||
|
# Wait for health check to detect (default: 5 seconds)
|
||||||
|
sleep 10
|
||||||
|
|
||||||
|
# Verify traffic only goes to remaining backends (external)
|
||||||
|
for i in $(seq 1 6); do
|
||||||
|
curl -s http://192.168.103.200/ | grep -o '<h1>[^<]*</h1>'
|
||||||
|
done
|
||||||
|
# Expected: only nginx-lb-01 and nginx-lb-02
|
||||||
|
|
||||||
|
# Restart the backend
|
||||||
|
incus start oc-node-01:nginx-lb-03
|
||||||
|
|
||||||
|
# Wait for health check to re-add
|
||||||
|
sleep 10
|
||||||
|
|
||||||
|
# Verify all three are back in rotation
|
||||||
|
for i in $(seq 1 6); do
|
||||||
|
curl -s http://192.168.103.200/ | grep -o '<h1>[^<]*</h1>'
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
### HAProxy failover
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Stop one HAProxy container
|
||||||
|
incus stop oc-node-01:ffsdn-haproxy-52-02
|
||||||
|
|
||||||
|
# Wait for OVN to detect the dead backend
|
||||||
|
sleep 15
|
||||||
|
|
||||||
|
# Some requests will succeed (routed to HAProxy 01), others may time out
|
||||||
|
curl -s --connect-timeout 10 http://192.168.103.200/
|
||||||
|
# Expected: response from one of the backends (via HAProxy 01)
|
||||||
|
|
||||||
|
# Restart
|
||||||
|
incus start oc-node-01:ffsdn-haproxy-52-02
|
||||||
|
```
|
||||||
|
|
||||||
|
**Note**: OVN load balancers do NOT have active health checking. When one
|
||||||
|
HAProxy instance is stopped, the OVN LB continues to send ~50% of
|
||||||
|
connections to the dead backend, causing those connections to time out.
|
||||||
|
After 10-15 seconds OVN may detect the dead backend, but failover is
|
||||||
|
not instant. For production use, consider Keepalived mode which provides
|
||||||
|
true active/standby VIP failover with VRRP.
|
||||||
|
|
||||||
|
## Monitoring
|
||||||
|
|
||||||
|
### Health and statistics
|
||||||
|
|
||||||
|
Each service has a Health & Stats view accessible via the Aether UI
|
||||||
|
(**Services** > **Health & Stats** button) or the session API:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -sSk -b "$COOKIE_JAR" \
|
||||||
|
-H "X-CSRF-Token: ${CSRF}" \
|
||||||
|
"${AETHER_URL}/haproxy/service/health-stats?cluster_id=52&service_name=web-http"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Statistics fields
|
||||||
|
|
||||||
|
**Backend server statistics** (from HAProxy health checks):
|
||||||
|
|
||||||
|
| Field | Description |
|
||||||
|
|-------|-------------|
|
||||||
|
| Status | `UP` (healthy) or `DOWN` (removed from rotation) |
|
||||||
|
| Sessions | Current active / maximum seen |
|
||||||
|
| Bytes In/Out | Total traffic to/from this backend |
|
||||||
|
| Weight | Configured load distribution weight |
|
||||||
|
| Last Check | Most recent health check result + response time (ms) |
|
||||||
|
|
||||||
|
**Frontend statistics** (combined for all services on the same port):
|
||||||
|
|
||||||
|
| Field | Description |
|
||||||
|
|-------|-------------|
|
||||||
|
| Current Sessions | Active client connections |
|
||||||
|
| Max Sessions | Highest concurrent sessions seen |
|
||||||
|
| Total Sessions | Cumulative since HAProxy started |
|
||||||
|
| Session Rate | New sessions per second |
|
||||||
|
| Bytes In/Out | Total client traffic |
|
||||||
|
|
||||||
|
**HAProxy node resources** (per container):
|
||||||
|
|
||||||
|
| Field | Description | Color thresholds |
|
||||||
|
|-------|-------------|------------------|
|
||||||
|
| CPU Usage | Current utilization % | Green <50%, Orange 50-79%, Red 80%+ |
|
||||||
|
| Memory Usage | Current utilization | Green <50%, Orange 50-79%, Red 80%+ |
|
||||||
|
|
||||||
|
## Keepalived mode
|
||||||
|
|
||||||
|
For clusters without OVN networking, Aether supports Keepalived-based
|
||||||
|
HAProxy deployment. Instead of an OVN load balancer, Keepalived provides
|
||||||
|
VIP failover between the two HAProxy instances using VRRP.
|
||||||
|
|
||||||
|
Key differences:
|
||||||
|
- **No OVN required** — works with bridge or macvlan networking
|
||||||
|
- **VRRP failover** — active/standby instead of OVN's active/active LB
|
||||||
|
- **Direct VIP** — the VIP floats between HAProxy instances directly
|
||||||
|
|
||||||
|
This mode is selected during infrastructure deployment when choosing a
|
||||||
|
non-OVN network. The service configuration is identical regardless of
|
||||||
|
the underlying HA mechanism.
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### Image build fails
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check build status
|
||||||
|
curl -sSk -b "$COOKIE_JAR" \
|
||||||
|
-H "X-CSRF-Token: ${CSRF}" \
|
||||||
|
"${AETHER_URL}/haproxy/image/build-status"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Common causes:**
|
||||||
|
- No internet access from the cluster (package downloads fail)
|
||||||
|
- Cluster not reachable from Aether
|
||||||
|
- Insufficient disk space for image build
|
||||||
|
|
||||||
|
**Fix:** Ensure the cluster nodes can reach the internet. Try building
|
||||||
|
on a different cluster if available.
|
||||||
|
|
||||||
|
### Infrastructure deployment fails
|
||||||
|
|
||||||
|
**"No Image" in cluster dropdown:** The HAProxy image hasn't been pushed
|
||||||
|
to this cluster. Build an image (Step 2), then push it to the cluster
|
||||||
|
and set it as current.
|
||||||
|
|
||||||
|
**Containers not starting:** Check the Incus cluster has sufficient
|
||||||
|
resources (CPU, memory) for two additional containers.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check container state
|
||||||
|
incus list oc-node-01: -c ns --format csv | grep ffsdn-haproxy
|
||||||
|
```
|
||||||
|
|
||||||
|
### HAProxy container not starting after deploy
|
||||||
|
|
||||||
|
Aether may deploy both containers but only start one. This was observed
|
||||||
|
with `ffsdn-haproxy-52-01` remaining in STOPPED state after deployment.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check both containers
|
||||||
|
incus list oc-node-01: -f compact -c nsN4 | grep ffsdn-haproxy
|
||||||
|
|
||||||
|
# Start the stopped container
|
||||||
|
incus start oc-node-01:ffsdn-haproxy-52-01
|
||||||
|
```
|
||||||
|
|
||||||
|
### Service not responding
|
||||||
|
|
||||||
|
**VIP not reachable from outside OVN ("Connection refused"):**
|
||||||
|
Aether auto-generates per-instance ACL rules that only allow ingress
|
||||||
|
from `source: 192.168.103.200` (the VIP address). External traffic
|
||||||
|
preserves the original client IP as source, which the ACL rejects.
|
||||||
|
Apply the ACL fix from Step 3:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Diagnose: check the ACL rules
|
||||||
|
incus network acl show oc-node-01:52-ffsdn-haproxy-52-01-aether-acl
|
||||||
|
|
||||||
|
# If the only ingress rule has source: 192.168.103.200, add a broader rule:
|
||||||
|
for n in 01 02; do
|
||||||
|
incus network acl rule add "oc-node-01:52-ffsdn-haproxy-52-${n}-aether-acl" \
|
||||||
|
ingress action=allow protocol=tcp destination_port=80,443 \
|
||||||
|
description="Allow external clients to reach HAProxy"
|
||||||
|
done
|
||||||
|
|
||||||
|
# Verify it works now:
|
||||||
|
curl -s http://192.168.103.200/
|
||||||
|
```
|
||||||
|
|
||||||
|
**Why internal access works but external doesn't:** OVN hairpin-SNATs
|
||||||
|
traffic that originates inside the OVN network and gets load-balanced
|
||||||
|
back into the same network (source becomes .200, matching the ACL).
|
||||||
|
External traffic doesn't get SNAT'd, so the original client IP doesn't
|
||||||
|
match the narrow ACL rule.
|
||||||
|
|
||||||
|
**Backends showing DOWN in health stats (HTTP mode):**
|
||||||
|
When using **HTTP mode**, Aether generates `option httpchk` which sends
|
||||||
|
an `OPTIONS /` request. Nginx returns `405 Not Allowed`, marking backends
|
||||||
|
DOWN. **Use TCP mode** for nginx backends, or check the HAProxy stats:
|
||||||
|
```bash
|
||||||
|
# Check backend health from HAProxy container
|
||||||
|
incus exec oc-node-01:ffsdn-haproxy-52-01 -- \
|
||||||
|
curl -s http://localhost:8404/stats\;csv | grep backend_web
|
||||||
|
|
||||||
|
# Verify direct backend connectivity
|
||||||
|
incus exec oc-node-01:ffsdn-haproxy-52-01 -- curl -s http://10.10.10.60/
|
||||||
|
```
|
||||||
|
|
||||||
|
### Traffic not reaching backends
|
||||||
|
|
||||||
|
Even when the service is created and HAProxy is running, traffic may not
|
||||||
|
flow if:
|
||||||
|
|
||||||
|
- **ACL rules blocking traffic** — Aether creates per-instance ACLs with
|
||||||
|
default-deny. If the ACLs don't cover the traffic pattern, it gets
|
||||||
|
rejected. Check with `incus network acl show <remote>:<acl-name>`.
|
||||||
|
- **HAProxy config not reloaded** — click **Reload Config** in the Aether
|
||||||
|
UI or wait for automatic propagation.
|
||||||
|
- **Wrong backend port** — verify the backends are actually listening on
|
||||||
|
the configured port (80 in our case).
|
||||||
|
|
||||||
|
Test connectivity directly from a HAProxy container:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
incus exec oc-node-01:ffsdn-haproxy-52-01 -- curl -s http://10.10.10.60/
|
||||||
|
```
|
||||||
|
|
||||||
|
### Sticky sessions not working
|
||||||
|
|
||||||
|
Sticky sessions only work with **HTTP** and **HTTPS Termination** modes.
|
||||||
|
They do NOT work with TCP or HTTPS Passthrough (HAProxy can't insert
|
||||||
|
cookies without inspecting HTTP content). Verify your service mode.
|
||||||
|
|
||||||
|
### Advanced options not taking effect
|
||||||
|
|
||||||
|
Advanced options (compression, HSTS, rate limiting, access control) only
|
||||||
|
apply to **HTTP** and **HTTPS Termination** modes. They have no effect on
|
||||||
|
TCP or HTTPS Passthrough services, as those operate at L4 without
|
||||||
|
inspecting HTTP content.
|
||||||
|
|
||||||
|
### Rate limiting blocking legitimate users
|
||||||
|
|
||||||
|
If users behind corporate NAT or VPN share a single IP, they collectively
|
||||||
|
count against the per-IP rate limit. A web page loading 20 images generates
|
||||||
|
21 requests. Start with higher limits (500+) and reduce based on monitoring.
|
||||||
|
|
||||||
|
### Session authentication issues
|
||||||
|
|
||||||
|
Aether's HAProxy endpoints use session authentication with CSRF protection.
|
||||||
|
**curl-based session auth is unreliable** — use Playwright browser automation
|
||||||
|
for reliable interaction (see `incusos/helpers/aether-browser`).
|
||||||
|
|
||||||
|
Common issues with curl-based auth:
|
||||||
|
1. **CSRF 403**: The CSRF token binding requires proper browser cookie handling.
|
||||||
|
Playwright handles this automatically.
|
||||||
|
2. **Login page at `/`**: The login form is at root (`/`), NOT `/login`.
|
||||||
|
`GET /login` returns 404. The form POSTs to `/login`.
|
||||||
|
3. **Session expiry**: Re-authenticate by re-running the Playwright login.
|
||||||
|
|
||||||
|
### Cleanup
|
||||||
|
|
||||||
|
To completely remove HAProxy infrastructure:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Via Aether UI: HAProxy > select cluster > Delete Infrastructure
|
||||||
|
|
||||||
|
# Via session API (requires CSRF token from Playwright):
|
||||||
|
curl -sSk -b "$COOKIE_JAR" \
|
||||||
|
-H "X-CSRF-Token: ${CSRF}" \
|
||||||
|
-H "Referer: ${AETHER_URL}/haproxy" \
|
||||||
|
-X DELETE "${AETHER_URL}/haproxy/infrastructure/52"
|
||||||
|
|
||||||
|
# Remove the OVN load balancer (created manually, not cleaned by Aether)
|
||||||
|
incus network load-balancer delete oc-node-01:net-prod 192.168.103.200
|
||||||
|
|
||||||
|
# Clean up test backends
|
||||||
|
for name in nginx-lb-01 nginx-lb-02 nginx-lb-03; do
|
||||||
|
incus delete "oc-node-01:${name}" --force
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
**Warning**: Deleting infrastructure stops all traffic immediately.
|
||||||
|
It removes both HAProxy containers, all VIPs, all services, and
|
||||||
|
associated firewall rules.
|
||||||
Loading…
Reference in New Issue