Add HAProxy LB guide, deploy script, Playwright helper; validate end-to-end
Tear down and fully re-deploy HAProxy infrastructure from scratch to validate the guide reads correctly. Key fixes from re-validation: - Fix session API field names: lb_vip (not vip), target_ip/target_port (not ip/port), health_check_enabled (not health_check) - Document that Aether does NOT create the OVN load balancer — must be created manually via incus network load-balancer - Fix internal VIP test: OVN LB doesn't hairpin, use localhost instead - Document OVN LB has no health checking (HAProxy failover not instant) - Add cleanup step for manually-created OVN LB - Add aether-browser Playwright helper and .mcp.json config - Add deploy-haproxy script, haproxy-lb rules, full guide Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
0100182dd8
commit
c047bc6ed4
|
|
@ -0,0 +1,125 @@
|
|||
---
|
||||
paths:
|
||||
- "incusos/deploy-haproxy"
|
||||
- "incusos/helpers/aether-browser"
|
||||
- "notes/haproxy-guide.md"
|
||||
---
|
||||
|
||||
# HAProxy Load Balancing (Aether-Managed)
|
||||
|
||||
## Overview
|
||||
|
||||
- Aether manages HAProxy deployment: image builds, HA pair, VIP management,
|
||||
L7 service configuration, health monitoring.
|
||||
- Lab cluster: oc-lab-cluster (ID: 52), OVN network: net-prod (10.10.10.0/24).
|
||||
- VIP range: 192.168.103.200-210 (from UPLINK).
|
||||
- `deploy-haproxy` script: `--deploy`, `--status`, `--cleanup`, `--doctor`.
|
||||
|
||||
## Use Playwright for Aether session auth, NOT curl
|
||||
|
||||
HAProxy management endpoints use session-authenticated routes with CSRF
|
||||
protection. **curl cannot reliably handle this** — use Playwright instead.
|
||||
|
||||
- **Playwright MCP**: `.mcp.json` configures `@playwright/mcp@latest`.
|
||||
When loaded, use `browser_navigate`, `browser_click`, `browser_fill` tools.
|
||||
- **Helper**: `incusos/helpers/aether-browser` for standalone automation.
|
||||
- **Login page**: `GET /` (root, NOT `/login`). Form action: `POST /login`.
|
||||
- **CSRF token**: hidden `<input name="csrf_token">` in HTML form, NOT cookie.
|
||||
|
||||
## Key UI element IDs (for Playwright automation)
|
||||
|
||||
| Element | ID | Purpose |
|
||||
|---------|-----|---------|
|
||||
| Cluster dropdown | `#clusterSelect` | Select cluster to manage |
|
||||
| Deploy mode | `#deployMode` | OVN or Keepalived |
|
||||
| OVN network | `#deployNetwork` | Select network (populates IP dropdowns) |
|
||||
| LB VIP select | `#deployLBVIPSelect` | Choose VIP from UPLINK range |
|
||||
| HP01 IP select | `#deployHP01IPSelect` | Choose HAProxy 01 IP |
|
||||
| HP02 IP select | `#deployHP02IPSelect` | Choose HAProxy 02 IP |
|
||||
| CPU limit | `#deployCPU` | Default: 2 |
|
||||
| Memory limit | `#deployMemory` | Default: 1 GB |
|
||||
| Service VIP | `#serviceVIP` | VIP by ID (e.g., value="41") |
|
||||
| Service mode | `#serviceMode` | tcp, http, https-passthrough, https-termination |
|
||||
| Service port | `#servicePort` | Listen port |
|
||||
| Service balance | `#serviceBalance` | roundrobin, leastconn, source, first |
|
||||
| Health check | `#serviceHealthCheck` | Checkbox, default: checked |
|
||||
| Backend name | `input[name="backend_name_N"]` | Nth backend name |
|
||||
| Backend IP | `input[name="backend_ip_N"]` | Nth backend IP |
|
||||
| Backend port | `input[name="backend_port_N"]` | Nth backend port |
|
||||
|
||||
Key JS functions: `showEditServiceModal(serviceId)`,
|
||||
`showServiceHealthStats(serviceId, name)`, `reloadHAProxyConfig()`,
|
||||
`showAddServiceModal()`, `showAddVIPModalWithVIPs()`.
|
||||
|
||||
## Key API routes (session-authenticated, use via Playwright page.evaluate)
|
||||
|
||||
| Route | Method | Purpose |
|
||||
|-------|--------|---------|
|
||||
| `/haproxy/infrastructure/{cluster_id}` | GET | Deployment state (JSON) |
|
||||
| `/haproxy/infrastructure/deploy` | POST | Deploy infrastructure |
|
||||
| `/haproxy/service` | POST | Create service |
|
||||
| `/haproxy/service/{id}` | PUT | Update service |
|
||||
| `/haproxy/service/{id}` | DELETE | Delete service |
|
||||
| `/haproxy/reload/{cluster_id}` | POST | Reload HAProxy config |
|
||||
| `/haproxy/infrastructure/{cluster_id}` | DELETE | Remove infrastructure |
|
||||
|
||||
## API field names (validated 2026-02-24)
|
||||
|
||||
**Deploy infrastructure** (`POST /haproxy/infrastructure/deploy`):
|
||||
`cluster_id` (int), `ovn_network`, `lb_vip` (NOT `vip`), `haproxy_01_ip`,
|
||||
`haproxy_02_ip`, `cpu_limit` (string), `memory_limit` (string, e.g. "1GB")
|
||||
|
||||
**Create service** (`POST /haproxy/service`):
|
||||
`vip_id` (int, NOT VIP address), `name`, `description`, `hostname`,
|
||||
`listen_port` (int), `mode`, `balance_method`, `health_check_enabled` (bool),
|
||||
`session_persistence` (bool), `backends` array with objects:
|
||||
`{ name, target_ip, target_port (int), weight (int) }`
|
||||
|
||||
## Container naming
|
||||
|
||||
`ffsdn-haproxy-{clusterID}-01` and `ffsdn-haproxy-{clusterID}-02`
|
||||
For cluster 52: `ffsdn-haproxy-52-01`, `ffsdn-haproxy-52-02`.
|
||||
|
||||
## Lab IP assignments (live, validated 2026-02-24)
|
||||
|
||||
| Component | IP |
|
||||
|-----------|-----|
|
||||
| VIP | 192.168.103.200 |
|
||||
| HAProxy 01 | 10.10.10.50 (oc-node-01) |
|
||||
| HAProxy 02 | 10.10.10.51 (oc-node-02) |
|
||||
| nginx-lb-01 | 10.10.10.60 |
|
||||
| nginx-lb-02 | 10.10.10.61 |
|
||||
| nginx-lb-03 | 10.10.10.62 |
|
||||
|
||||
## Validated gotchas
|
||||
|
||||
- **OVN LB not created by Aether**: Aether deploys HAProxy containers and
|
||||
ACLs but does NOT create the OVN load balancer (`incus network load-balancer`).
|
||||
You must create it manually after deployment. See guide Step 3.
|
||||
- **HTTP mode health checks**: Aether generates `option httpchk` (sends
|
||||
`OPTIONS /`). Nginx returns 405 → backends marked DOWN. **Use TCP mode**
|
||||
for nginx backends, or configure nginx to accept OPTIONS.
|
||||
- **HP-01 may not auto-start**: After deploy, `ffsdn-haproxy-52-01` was
|
||||
sometimes found Stopped. Check both containers post-deploy. Not always
|
||||
reproducible — both started on the 2026-02-24 re-validation.
|
||||
- **OVN VIP external access**: Aether ACLs only allow ingress from
|
||||
`source: VIP_IP`. External traffic preserves original client IP → rejected.
|
||||
**Fix**: add broad ingress rule to each HAProxy ACL:
|
||||
`incus network acl rule add <remote>:<acl> ingress action=allow protocol=tcp destination_port=80,443`
|
||||
Must re-apply after infrastructure redeploy.
|
||||
- **Internal VIP not reachable**: The manually-created OVN LB handles
|
||||
external-to-internal traffic but does NOT hairpin for internal-to-VIP
|
||||
requests from the same logical switch. Use HAProxy localhost for internal tests.
|
||||
- **OVN LB has no health checking**: When one HAProxy instance is stopped,
|
||||
the OVN LB continues routing ~50% of connections to the dead backend,
|
||||
causing timeouts. Failover is not instant. Consider Keepalived mode for
|
||||
production deployments requiring true active/standby failover.
|
||||
- **Static IPs on OVN**: Use `ipv4.address` device key on the NIC device,
|
||||
not systemd-networkd: `incus launch ... -d eth0,ipv4.address=10.10.10.60`.
|
||||
|
||||
## Key differences from deploy-awx
|
||||
|
||||
- **No VM creation**: HAProxy uses Aether-managed containers, not a standalone VM.
|
||||
- **Playwright auth**: Uses browser automation, not JWT or basic auth.
|
||||
- **No --heal action**: HAProxy is fully Aether-managed. Recovery = `--cleanup` + `--deploy`.
|
||||
- **Backends via incus CLI**: Test nginx backends created directly with `ipv4.address` device key.
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
{
|
||||
"mcpServers": {
|
||||
"playwright": {
|
||||
"command": "npx",
|
||||
"args": ["@playwright/mcp@latest"]
|
||||
}
|
||||
}
|
||||
}
|
||||
41
CLAUDE.md
41
CLAUDE.md
|
|
@ -26,10 +26,12 @@ incus-contrib/
|
|||
│ ├── incusos-seed # Seed archive generator (Linux + macOS)
|
||||
│ ├── incusos-proxmox # Declarative Proxmox VM deployment + lab lifecycle
|
||||
│ ├── deploy-awx # AWX deployment + management on Incus cluster
|
||||
│ ├── deploy-haproxy # HAProxy LB deployment + management via Aether
|
||||
│ ├── awx-manifests/ # K8s manifests for AWX Operator + instance
|
||||
│ ├── helpers/
|
||||
│ │ ├── proxmox-screenshot # VMID -> PNG console screenshot
|
||||
│ │ └── proxmox-api # Authenticated API calls (handles ! in token)
|
||||
│ │ ├── proxmox-api # Authenticated API calls (handles ! in token)
|
||||
│ │ └── aether-browser # Playwright browser automation for Aether web UI
|
||||
│ ├── lab-test # Guided lab validation (12 test phases)
|
||||
│ ├── observe-deploy # Single-VM deploy with console screenshots
|
||||
│ ├── proxmox.yaml # Proxmox connection (gitignored)
|
||||
|
|
@ -95,6 +97,42 @@ curl -sk http://192.168.102.161:30080/api/v2/jobs/{JOB_ID}/stdout/?format=txt \
|
|||
-H "Authorization: Bearer $AWX_TOKEN"
|
||||
```
|
||||
|
||||
### Aether browser automation (Playwright)
|
||||
|
||||
Many Aether features (HAProxy management, blueprints, deploys) are **not in
|
||||
the JWT API** — they use session-authenticated routes with CSRF protection
|
||||
that curl cannot handle reliably. **USE PLAYWRIGHT** for these interactions.
|
||||
|
||||
**MCP server**: Configured in `.mcp.json` — provides `browser_navigate`,
|
||||
`browser_click`, `browser_fill`, `browser_screenshot` etc. as tools when
|
||||
the Playwright MCP server is running. Prefer MCP tools when available.
|
||||
|
||||
**Helper script**: `incusos/helpers/aether-browser` provides standalone
|
||||
Playwright automation when MCP tools aren't loaded:
|
||||
```bash
|
||||
source env # loads AETHER_ADMIN_PASSWORD
|
||||
NODE_PATH=~/node_modules node incusos/helpers/aether-browser <action> [args]
|
||||
```
|
||||
|
||||
Actions:
|
||||
- `login` — authenticate and save session cookies to `/tmp/aether-cookies.json`
|
||||
- `screenshot <path>` — navigate to path, take screenshot
|
||||
- `navigate <path>` — navigate and show page title/URL
|
||||
- `haproxy-status` — screenshot HAProxy management page
|
||||
- `haproxy-images` — list built HAProxy images via session API
|
||||
- `api-get <endpoint>` — authenticated GET using browser session
|
||||
- `api-post <endpoint> <json>` — authenticated POST with CSRF handling
|
||||
- `eval <js>` — evaluate JavaScript on the current page
|
||||
|
||||
**When to use what**:
|
||||
- JWT API (`/api/*` endpoints): use `curl` with bearer token
|
||||
- Session-authenticated routes (`/haproxy/*`, UI actions): use Playwright
|
||||
- Never use curl for session auth — CSRF protection requires browser cookies
|
||||
|
||||
**Key technical detail**: Aether login is at `/` (root), NOT `/login`.
|
||||
The form POSTs to `/login`. CSRF token is in a hidden `<input>` field,
|
||||
not in cookies.
|
||||
|
||||
## Critical safety rules
|
||||
|
||||
- **Proxmox SSH is screenshots-only.** Full rules in `.claude/rules/proxmox-ssh-rules.md`.
|
||||
|
|
@ -146,6 +184,7 @@ which files are being edited:
|
|||
| `operations-center.md` | OC guide, incusos-proxmox, OC example configs |
|
||||
| `networking-storage.md` | networking, storage, migration, UTM guides |
|
||||
| `awx-integration.md` | ansible/, deploy-awx, awx-manifests, AWX guide |
|
||||
| `haproxy-lb.md` | deploy-haproxy, HAProxy guide |
|
||||
| `proxmox-ssh-rules.md` | **Always loaded** (no paths filter) |
|
||||
| `lab-infrastructure.md` | incusos-proxmox, lab-test, examples |
|
||||
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,306 @@
|
|||
#!/usr/bin/env node
|
||||
// aether-browser: Playwright helper for Aether web UI automation
|
||||
// Usage: aether-browser <action> [args...]
|
||||
//
|
||||
// Actions:
|
||||
// login - Login and save session cookies
|
||||
// screenshot <path> - Take screenshot of current page
|
||||
// haproxy-status - Check HAProxy infrastructure status
|
||||
// haproxy-images - List HAProxy images
|
||||
// navigate <url> - Navigate to URL and return page content
|
||||
// eval <js> - Evaluate JavaScript on current page
|
||||
//
|
||||
// Environment:
|
||||
// AETHER_URL - Aether base URL (default: https://192.168.102.160:8443)
|
||||
// AETHER_ADMIN_PASSWORD - Admin password (required)
|
||||
// AETHER_COOKIE_FILE - Cookie persistence file (default: /tmp/aether-cookies.json)
|
||||
// HEADLESS - Set to "false" for visible browser (default: true)
|
||||
|
||||
const { chromium } = require('playwright-core');
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
|
||||
const AETHER_URL = process.env.AETHER_URL || 'https://192.168.102.160:8443';
|
||||
const PASSWORD = process.env.AETHER_ADMIN_PASSWORD;
|
||||
const COOKIE_FILE = process.env.AETHER_COOKIE_FILE || '/tmp/aether-cookies.json';
|
||||
const HEADLESS = process.env.HEADLESS !== 'false';
|
||||
const SCREENSHOT_DIR = '/tmp';
|
||||
|
||||
if (!PASSWORD) {
|
||||
console.error('ERROR: AETHER_ADMIN_PASSWORD not set. Source the env file first.');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
async function getBrowser() {
|
||||
// Use Playwright's bundled Chromium
|
||||
const browserPath = require('playwright-core').chromium.executablePath();
|
||||
return await chromium.launch({
|
||||
headless: HEADLESS,
|
||||
executablePath: browserPath,
|
||||
args: ['--ignore-certificate-errors', '--no-sandbox'],
|
||||
});
|
||||
}
|
||||
|
||||
async function getContextWithCookies(browser) {
|
||||
const context = await browser.newContext({
|
||||
ignoreHTTPSErrors: true,
|
||||
viewport: { width: 1280, height: 800 },
|
||||
});
|
||||
|
||||
// Restore cookies if available
|
||||
if (fs.existsSync(COOKIE_FILE)) {
|
||||
try {
|
||||
const cookies = JSON.parse(fs.readFileSync(COOKIE_FILE, 'utf8'));
|
||||
await context.addCookies(cookies);
|
||||
} catch (e) {
|
||||
// Stale cookies, will re-login
|
||||
}
|
||||
}
|
||||
|
||||
return context;
|
||||
}
|
||||
|
||||
async function saveCookies(context) {
|
||||
const cookies = await context.cookies();
|
||||
fs.writeFileSync(COOKIE_FILE, JSON.stringify(cookies, null, 2));
|
||||
}
|
||||
|
||||
async function login(page) {
|
||||
console.log(`Navigating to ${AETHER_URL}/...`);
|
||||
await page.goto(`${AETHER_URL}/`, { waitUntil: 'networkidle' });
|
||||
|
||||
// Check if already logged in
|
||||
const url = page.url();
|
||||
if (!url.includes('login') && !await page.locator('input[name="csrf_token"]').count()) {
|
||||
console.log('Already logged in (session cookies valid).');
|
||||
return true;
|
||||
}
|
||||
|
||||
console.log('Login page detected. Filling credentials...');
|
||||
|
||||
// Fill the login form
|
||||
await page.fill('input[name="username"]', 'admin');
|
||||
await page.fill('input[name="password"]', PASSWORD);
|
||||
|
||||
// Submit
|
||||
await page.click('button[type="submit"], input[type="submit"]');
|
||||
|
||||
// Wait for navigation after login
|
||||
await page.waitForLoadState('networkidle');
|
||||
|
||||
// Check success
|
||||
const afterUrl = page.url();
|
||||
if (afterUrl.includes('login')) {
|
||||
console.error('ERROR: Login failed - still on login page.');
|
||||
await page.screenshot({ path: `${SCREENSHOT_DIR}/aether-login-fail.png` });
|
||||
console.error(`Screenshot saved to ${SCREENSHOT_DIR}/aether-login-fail.png`);
|
||||
return false;
|
||||
}
|
||||
|
||||
console.log(`Login successful. Redirected to: ${afterUrl}`);
|
||||
return true;
|
||||
}
|
||||
|
||||
async function doAction(action, args) {
|
||||
const browser = await getBrowser();
|
||||
const context = await getContextWithCookies(browser);
|
||||
const page = await context.newPage();
|
||||
|
||||
try {
|
||||
switch (action) {
|
||||
case 'login': {
|
||||
const success = await login(page);
|
||||
if (success) {
|
||||
await saveCookies(context);
|
||||
console.log(`Session cookies saved to ${COOKIE_FILE}`);
|
||||
}
|
||||
break;
|
||||
}
|
||||
|
||||
case 'screenshot': {
|
||||
const loginOk = await login(page);
|
||||
if (!loginOk) break;
|
||||
await saveCookies(context);
|
||||
|
||||
const targetUrl = args[0] || '/';
|
||||
if (targetUrl.startsWith('/')) {
|
||||
await page.goto(`${AETHER_URL}${targetUrl}`, { waitUntil: 'networkidle' });
|
||||
}
|
||||
|
||||
const outPath = args[1] || `${SCREENSHOT_DIR}/aether-screenshot.png`;
|
||||
await page.screenshot({ path: outPath, fullPage: true });
|
||||
console.log(`Screenshot saved to ${outPath}`);
|
||||
break;
|
||||
}
|
||||
|
||||
case 'haproxy-status': {
|
||||
const loginOk = await login(page);
|
||||
if (!loginOk) break;
|
||||
await saveCookies(context);
|
||||
|
||||
// Navigate to HAProxy infrastructure page
|
||||
console.log('Navigating to HAProxy management...');
|
||||
await page.goto(`${AETHER_URL}/haproxy`, { waitUntil: 'networkidle' });
|
||||
await page.screenshot({ path: `${SCREENSHOT_DIR}/aether-haproxy.png`, fullPage: true });
|
||||
console.log(`Screenshot: ${SCREENSHOT_DIR}/aether-haproxy.png`);
|
||||
|
||||
// Try to extract status info from the page
|
||||
const content = await page.textContent('body');
|
||||
// Print relevant sections
|
||||
const lines = content.split('\n').filter(l => l.trim());
|
||||
console.log('\n--- Page content (filtered) ---');
|
||||
for (const line of lines.slice(0, 50)) {
|
||||
console.log(line.trim());
|
||||
}
|
||||
break;
|
||||
}
|
||||
|
||||
case 'haproxy-images': {
|
||||
const loginOk = await login(page);
|
||||
if (!loginOk) break;
|
||||
await saveCookies(context);
|
||||
|
||||
console.log('Fetching HAProxy images...');
|
||||
// Use the session to make an API call
|
||||
const response = await page.evaluate(async () => {
|
||||
const resp = await fetch('/haproxy/images', {
|
||||
credentials: 'same-origin',
|
||||
});
|
||||
return { status: resp.status, body: await resp.text() };
|
||||
});
|
||||
console.log(`Status: ${response.status}`);
|
||||
console.log(response.body);
|
||||
break;
|
||||
}
|
||||
|
||||
case 'navigate': {
|
||||
const loginOk = await login(page);
|
||||
if (!loginOk) break;
|
||||
await saveCookies(context);
|
||||
|
||||
const navUrl = args[0] || '/';
|
||||
const fullUrl = navUrl.startsWith('http') ? navUrl : `${AETHER_URL}${navUrl}`;
|
||||
console.log(`Navigating to ${fullUrl}...`);
|
||||
await page.goto(fullUrl, { waitUntil: 'networkidle' });
|
||||
|
||||
const outPath = `${SCREENSHOT_DIR}/aether-navigate.png`;
|
||||
await page.screenshot({ path: outPath, fullPage: true });
|
||||
console.log(`Screenshot: ${outPath}`);
|
||||
|
||||
const title = await page.title();
|
||||
console.log(`Title: ${title}`);
|
||||
console.log(`URL: ${page.url()}`);
|
||||
break;
|
||||
}
|
||||
|
||||
case 'api-get': {
|
||||
const loginOk = await login(page);
|
||||
if (!loginOk) break;
|
||||
await saveCookies(context);
|
||||
|
||||
const endpoint = args[0];
|
||||
if (!endpoint) {
|
||||
console.error('Usage: aether-browser api-get <endpoint>');
|
||||
break;
|
||||
}
|
||||
|
||||
const response = await page.evaluate(async (url) => {
|
||||
const resp = await fetch(url, { credentials: 'same-origin' });
|
||||
const text = await resp.text();
|
||||
return { status: resp.status, contentType: resp.headers.get('content-type'), body: text };
|
||||
}, endpoint);
|
||||
|
||||
console.log(`GET ${endpoint} -> ${response.status}`);
|
||||
if (response.contentType && response.contentType.includes('json')) {
|
||||
try {
|
||||
console.log(JSON.stringify(JSON.parse(response.body), null, 2));
|
||||
} catch {
|
||||
console.log(response.body);
|
||||
}
|
||||
} else {
|
||||
console.log(response.body);
|
||||
}
|
||||
break;
|
||||
}
|
||||
|
||||
case 'api-post': {
|
||||
const loginOk = await login(page);
|
||||
if (!loginOk) break;
|
||||
await saveCookies(context);
|
||||
|
||||
const endpoint = args[0];
|
||||
const bodyJson = args[1];
|
||||
if (!endpoint || !bodyJson) {
|
||||
console.error('Usage: aether-browser api-post <endpoint> <json-body>');
|
||||
break;
|
||||
}
|
||||
|
||||
// Get CSRF token from meta tag or cookie
|
||||
const csrfToken = await page.evaluate(() => {
|
||||
const meta = document.querySelector('meta[name="csrf-token"]');
|
||||
if (meta) return meta.getAttribute('content');
|
||||
// Try from cookie
|
||||
const match = document.cookie.match(/csrf_token=([^;]+)/);
|
||||
return match ? match[1] : '';
|
||||
});
|
||||
|
||||
const response = await page.evaluate(async ({ url, body, csrf }) => {
|
||||
const headers = {
|
||||
'Content-Type': 'application/json',
|
||||
};
|
||||
if (csrf) {
|
||||
headers['X-CSRFToken'] = csrf;
|
||||
headers['X-CSRF-Token'] = csrf;
|
||||
}
|
||||
const resp = await fetch(url, {
|
||||
method: 'POST',
|
||||
credentials: 'same-origin',
|
||||
headers,
|
||||
body,
|
||||
});
|
||||
const text = await resp.text();
|
||||
return { status: resp.status, body: text };
|
||||
}, { url: endpoint, body: bodyJson, csrf: csrfToken });
|
||||
|
||||
console.log(`POST ${endpoint} -> ${response.status}`);
|
||||
try {
|
||||
console.log(JSON.stringify(JSON.parse(response.body), null, 2));
|
||||
} catch {
|
||||
console.log(response.body);
|
||||
}
|
||||
break;
|
||||
}
|
||||
|
||||
case 'eval': {
|
||||
const loginOk = await login(page);
|
||||
if (!loginOk) break;
|
||||
await saveCookies(context);
|
||||
|
||||
const js = args.join(' ');
|
||||
const result = await page.evaluate(js);
|
||||
console.log(JSON.stringify(result, null, 2));
|
||||
break;
|
||||
}
|
||||
|
||||
default:
|
||||
console.error(`Unknown action: ${action}`);
|
||||
console.error('Actions: login, screenshot, haproxy-status, haproxy-images, navigate, api-get, api-post, eval');
|
||||
process.exit(1);
|
||||
}
|
||||
} finally {
|
||||
await browser.close();
|
||||
}
|
||||
}
|
||||
|
||||
// Main
|
||||
const [action, ...args] = process.argv.slice(2);
|
||||
if (!action) {
|
||||
console.error('Usage: aether-browser <action> [args...]');
|
||||
console.error('Actions: login, screenshot, haproxy-status, haproxy-images, navigate, api-get, api-post, eval');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
doAction(action, args).catch(err => {
|
||||
console.error('Fatal error:', err.message);
|
||||
process.exit(1);
|
||||
});
|
||||
|
|
@ -0,0 +1,780 @@
|
|||
# HAProxy Load Balancing — Aether-Managed HA Proxy on OVN
|
||||
|
||||
Aether includes a full HAProxy management system that deploys a highly
|
||||
available load balancer pair on OVN networks. It handles image builds,
|
||||
HA pair deployment, VIP management, L7 service configuration, and health
|
||||
monitoring — all through its web UI and session-authenticated API.
|
||||
|
||||
This guide covers deploying HAProxy infrastructure on the lab cluster,
|
||||
creating test backends, configuring services, and verifying load balancing.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Client / LAN traffic
|
||||
|
|
||||
v
|
||||
+---------------------------+
|
||||
| UPLINK VIP Address |
|
||||
| 192.168.103.200 |
|
||||
+---------------------------+
|
||||
|
|
||||
v
|
||||
+---------------------------+
|
||||
| OVN Load Balancer |
|
||||
| (distributes to HA |
|
||||
| pair for failover) |
|
||||
+---------------------------+
|
||||
/ \
|
||||
/ \
|
||||
v v
|
||||
+-----------------+ +-----------------+
|
||||
| HAProxy 01 | | HAProxy 02 |
|
||||
| 10.10.10.50 | | 10.10.10.51 |
|
||||
| (container) | | (container) |
|
||||
+-----------------+ +-----------------+
|
||||
\ | /
|
||||
\ | /
|
||||
v v v
|
||||
+----------+ +----------+ +----------+
|
||||
| nginx-01 | | nginx-02 | | nginx-03 |
|
||||
| .60 :80 | | .61 :80 | | .62 :80 |
|
||||
+----------+ +----------+ +----------+
|
||||
(backend servers on net-prod)
|
||||
```
|
||||
|
||||
### OVN native LB vs HAProxy
|
||||
|
||||
| Feature | OVN native LB | Aether HAProxy |
|
||||
|---------|---------------|----------------|
|
||||
| Layer | L4 (TCP/UDP) | L4 + L7 (HTTP/HTTPS) |
|
||||
| Health checks | None | HTTP, TCP, configurable interval |
|
||||
| SSL termination | No | Yes (HTTPS Termination mode) |
|
||||
| Sticky sessions | No | Yes (cookie-based) |
|
||||
| Load balancing | Connection hash | Round robin, least conn, source hash, first available |
|
||||
| SNI routing | No | Yes (multiple domains on same VIP:port) |
|
||||
| Compression | No | Yes (gzip for text content) |
|
||||
| Rate limiting | No | Yes (per-IP) |
|
||||
| Management | `incus network load-balancer` CLI | Aether UI + session API |
|
||||
| HA | Single OVN LB (distributed) | HA pair with OVN LB failover |
|
||||
|
||||
Use OVN native LB for simple L4 distribution. Use HAProxy when you need
|
||||
health checks, L7 features, SSL, sticky sessions, or advanced traffic management.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Incus cluster registered in Aether with OVN networking configured
|
||||
- OVN network with a physical UPLINK (for VIP addresses)
|
||||
- Available IP addresses on the OVN network for HAProxy instances
|
||||
- Available VIP address(es) from the UPLINK range
|
||||
- Internet access from the cluster (for HAProxy image builds)
|
||||
- `incus` CLI configured with cluster remote
|
||||
- Aether admin access (Full Admin or LB Admin role)
|
||||
|
||||
### Resource requirements
|
||||
|
||||
| Resource | Per HAProxy instance | Notes |
|
||||
|----------|---------------------|-------|
|
||||
| CPU | 2 cores (default) | HAProxy is very CPU-efficient |
|
||||
| Memory | 1 GiB (default) | Increase for many concurrent connections |
|
||||
| Disk | Minimal | Container image, no persistent storage needed |
|
||||
|
||||
## Lab details
|
||||
|
||||
| Setting | Value |
|
||||
|---------|-------|
|
||||
| Cluster | oc-lab-cluster (ID: 52) |
|
||||
| OVN network | net-prod (10.10.10.0/24) |
|
||||
| UPLINK | UPLINK (physical) |
|
||||
| VIP range | 192.168.103.200-210 |
|
||||
| HAProxy 01 IP | 10.10.10.50 |
|
||||
| HAProxy 02 IP | 10.10.10.51 |
|
||||
| Backend IPs | 10.10.10.60, 10.10.10.61, 10.10.10.62 |
|
||||
| HAProxy containers | ffsdn-haproxy-52-01, ffsdn-haproxy-52-02 |
|
||||
| Aether URL | https://192.168.102.160:8443 |
|
||||
| Cluster remote | oc-node-01 |
|
||||
|
||||
## Automated path
|
||||
|
||||
The `incusos/deploy-haproxy` script automates the entire deployment:
|
||||
|
||||
```bash
|
||||
# Check prerequisites
|
||||
deploy-haproxy --doctor
|
||||
|
||||
# Preview deployment
|
||||
deploy-haproxy --deploy --dry-run
|
||||
|
||||
# Full deployment (backends + image + infrastructure + service)
|
||||
deploy-haproxy --deploy
|
||||
|
||||
# Check health
|
||||
deploy-haproxy --status
|
||||
|
||||
# Clean removal
|
||||
deploy-haproxy --cleanup
|
||||
```
|
||||
|
||||
The manual steps below are for reference, troubleshooting, and understanding
|
||||
what the script does.
|
||||
|
||||
## Step 1: Deploy test backends
|
||||
|
||||
Create 3 nginx containers on net-prod to serve as load-balanced backends.
|
||||
Each gets a unique index page so we can verify traffic distribution.
|
||||
|
||||
**Important**: The backend IPs (10.10.10.60-62) must not already be in use
|
||||
on the OVN network. Check the Aether UI (HAProxy > select cluster) for
|
||||
used IPs before deploying, or use `deploy-haproxy --doctor` which checks
|
||||
for you.
|
||||
|
||||
```bash
|
||||
REMOTE="oc-node-01"
|
||||
|
||||
for i in 60 61 62; do
|
||||
NAME="nginx-lb-$(printf '%02d' $((i - 59)))"
|
||||
IP="10.10.10.${i}"
|
||||
|
||||
# Launch container on OVN network with static IP via device key
|
||||
incus launch images:debian/12 "${REMOTE}:${NAME}" \
|
||||
--network net-prod \
|
||||
--storage local \
|
||||
-d root,size=2GiB \
|
||||
-d eth0,ipv4.address="${IP}" \
|
||||
-c limits.cpu=1 \
|
||||
-c limits.memory=256MiB
|
||||
|
||||
# Wait for container agent
|
||||
echo "Waiting for ${NAME}..."
|
||||
while ! incus exec "${REMOTE}:${NAME}" -- true 2>/dev/null; do
|
||||
sleep 2
|
||||
done
|
||||
|
||||
# Install nginx + curl, create unique index page
|
||||
incus exec "${REMOTE}:${NAME}" -- bash -c "
|
||||
export DEBIAN_FRONTEND=noninteractive
|
||||
apt-get update -qq
|
||||
apt-get install -y -qq nginx curl 2>&1 | tail -1
|
||||
|
||||
cat > /var/www/html/index.html << HTML
|
||||
<html><body>
|
||||
<h1>Backend: ${NAME}</h1>
|
||||
<p>IP: ${IP}</p>
|
||||
</body></html>
|
||||
HTML
|
||||
|
||||
systemctl enable nginx
|
||||
systemctl start nginx
|
||||
"
|
||||
|
||||
echo "Backend ${NAME} deployed at ${IP}"
|
||||
done
|
||||
```
|
||||
|
||||
Verify all backends respond (curling from one backend to the others
|
||||
confirms OVN cross-container connectivity):
|
||||
|
||||
```bash
|
||||
# Wait for static IPs to take effect
|
||||
sleep 5
|
||||
|
||||
for i in 60 61 62; do
|
||||
incus exec "${REMOTE}:nginx-lb-01" -- \
|
||||
curl -s --connect-timeout 3 "http://10.10.10.${i}/" \
|
||||
| grep -o 'Backend: [^<]*'
|
||||
done
|
||||
# Expected:
|
||||
# Backend: nginx-lb-01
|
||||
# Backend: nginx-lb-02
|
||||
# Backend: nginx-lb-03
|
||||
```
|
||||
|
||||
If curl fails, the static IP may not have taken effect yet. Verify with
|
||||
`incus exec "${REMOTE}:nginx-lb-01" -- ip addr show eth0`.
|
||||
|
||||
## Step 2: Build HAProxy image
|
||||
|
||||
Aether builds a base HAProxy container image and stores it internally.
|
||||
The image must be pushed to each target cluster before infrastructure
|
||||
can be deployed.
|
||||
|
||||
**Note**: The HAProxy management features are not in Aether's JWT API
|
||||
(`/api/swagger.yaml`). They use session-authenticated routes discovered
|
||||
from the UI JavaScript. The script uses session auth automatically.
|
||||
|
||||
### Via Aether UI
|
||||
|
||||
1. Navigate to **HAProxy** in the main menu
|
||||
2. In **HAProxy Base Images**, click **Build New Image**
|
||||
3. Select **Build on Cluster**: `oc-lab-cluster`
|
||||
4. Enter **Version**: `1.0.0`
|
||||
5. Click **Build Image**
|
||||
6. Wait 5-10 minutes for the build to complete
|
||||
7. Click **Push to Cluster** and select `oc-lab-cluster`
|
||||
8. Click **Set Current** on the new image
|
||||
|
||||
### Via session API
|
||||
|
||||
The build process uses session cookies + CSRF token authentication:
|
||||
|
||||
```bash
|
||||
AETHER_URL="https://192.168.102.160:8443"
|
||||
COOKIE_JAR="/tmp/aether-cookies.txt"
|
||||
|
||||
# Step 1: Get login page and extract CSRF token from hidden form field
|
||||
# Note: the login form is at "/" (root), NOT "/login". GET /login returns 404.
|
||||
CSRF=$(curl -sSk -c "$COOKIE_JAR" "${AETHER_URL}/" \
|
||||
| grep -o 'name="csrf_token" value="[^"]*"' \
|
||||
| sed 's/.*value="//;s/"//')
|
||||
|
||||
# Step 2: Login with CSRF token (form-urlencoded, NOT JSON)
|
||||
curl -sSk -b "$COOKIE_JAR" -c "$COOKIE_JAR" \
|
||||
-X POST "${AETHER_URL}/login" \
|
||||
-d "csrf_token=${CSRF}&username=admin&password=${AETHER_ADMIN_PASSWORD}" \
|
||||
-L -o /dev/null
|
||||
|
||||
# Step 3: Get fresh CSRF token for API calls
|
||||
CSRF=$(curl -sSk -b "$COOKIE_JAR" "${AETHER_URL}/haproxy" \
|
||||
| grep -o 'name="csrf_token" value="[^"]*"' \
|
||||
| sed 's/.*value="//;s/"//')
|
||||
|
||||
# Step 4: Build image
|
||||
curl -sSk -b "$COOKIE_JAR" \
|
||||
-H "X-CSRF-Token: ${CSRF}" \
|
||||
-H "Referer: ${AETHER_URL}/haproxy" \
|
||||
-X POST "${AETHER_URL}/haproxy/image/build" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"cluster_id": 52, "version": "1.0.0"}'
|
||||
|
||||
# Step 5: Poll build status until complete
|
||||
while true; do
|
||||
STATUS=$(curl -sSk -b "$COOKIE_JAR" \
|
||||
-H "X-CSRF-Token: ${CSRF}" \
|
||||
"${AETHER_URL}/haproxy/image/build-status" \
|
||||
| python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('status','unknown'))")
|
||||
echo "Build status: $STATUS"
|
||||
[[ "$STATUS" == "completed" || "$STATUS" == "failed" ]] && break
|
||||
sleep 10
|
||||
done
|
||||
|
||||
# Step 6: Get image ID and push to cluster
|
||||
IMAGE_ID=$(curl -sSk -b "$COOKIE_JAR" \
|
||||
-H "X-CSRF-Token: ${CSRF}" \
|
||||
"${AETHER_URL}/haproxy/images" \
|
||||
| python3 -c "import sys,json; imgs=json.load(sys.stdin); print(imgs[-1]['id'] if imgs else '')")
|
||||
|
||||
curl -sSk -b "$COOKIE_JAR" \
|
||||
-H "X-CSRF-Token: ${CSRF}" \
|
||||
-H "Referer: ${AETHER_URL}/haproxy" \
|
||||
-X POST "${AETHER_URL}/haproxy/image/push" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "{\"image_id\": ${IMAGE_ID}, \"cluster_id\": 52}"
|
||||
|
||||
# Step 7: Set as current version
|
||||
curl -sSk -b "$COOKIE_JAR" \
|
||||
-H "X-CSRF-Token: ${CSRF}" \
|
||||
-H "Referer: ${AETHER_URL}/haproxy" \
|
||||
-X POST "${AETHER_URL}/haproxy/image/set-current" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "{\"image_id\": ${IMAGE_ID}}"
|
||||
```
|
||||
|
||||
**Critical**: The CSRF token comes from a hidden HTML form field
|
||||
(`<input name="csrf_token" value="...">`), NOT from the cookie value.
|
||||
The cookie contains the session ID; the form field contains the CSRF token.
|
||||
|
||||
## Step 3: Deploy HAProxy infrastructure
|
||||
|
||||
Deploy the HA pair on the cluster, then create the OVN load balancer.
|
||||
|
||||
### Via Aether UI
|
||||
|
||||
1. Navigate to **HAProxy** > select **oc-lab-cluster** from dropdown
|
||||
2. Click **Deploy HAProxy Infrastructure**
|
||||
3. Fill in:
|
||||
- **OVN Network**: `net-prod`
|
||||
- **Load Balancer VIP**: `192.168.103.200` (from UPLINK range dropdown)
|
||||
- **HAProxy 01 IP**: `10.10.10.50`
|
||||
- **HAProxy 02 IP**: `10.10.10.51`
|
||||
- **CPU Limit**: `2`
|
||||
- **Memory Limit**: `1GB`
|
||||
4. Click **Deploy**
|
||||
|
||||
### Via session API
|
||||
|
||||
```bash
|
||||
curl -sSk -b "$COOKIE_JAR" \
|
||||
-H "X-CSRF-Token: ${CSRF}" \
|
||||
-H "Referer: ${AETHER_URL}/haproxy" \
|
||||
-X POST "${AETHER_URL}/haproxy/infrastructure/deploy" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"cluster_id": 52,
|
||||
"ovn_network": "net-prod",
|
||||
"lb_vip": "192.168.103.200",
|
||||
"haproxy_01_ip": "10.10.10.50",
|
||||
"haproxy_02_ip": "10.10.10.51",
|
||||
"cpu_limit": "2",
|
||||
"memory_limit": "1GB"
|
||||
}'
|
||||
```
|
||||
|
||||
This creates:
|
||||
- Two containers: `ffsdn-haproxy-52-01` and `ffsdn-haproxy-52-02`
|
||||
- Per-instance ACL rules allowing traffic from the VIP
|
||||
- A default VIP entry at 192.168.103.200
|
||||
|
||||
**Important**: Aether does NOT create the OVN load balancer automatically.
|
||||
You must create it manually after infrastructure deployment:
|
||||
|
||||
```bash
|
||||
REMOTE="oc-node-01"
|
||||
|
||||
# Create OVN load balancer for the VIP
|
||||
incus network load-balancer create "${REMOTE}:net-prod" 192.168.103.200 \
|
||||
--description "FFSDN HAProxy Load Balancer"
|
||||
|
||||
# Add both HAProxy instances as backends
|
||||
incus network load-balancer backend add "${REMOTE}:net-prod" 192.168.103.200 \
|
||||
haproxy-01 10.10.10.50 80
|
||||
incus network load-balancer backend add "${REMOTE}:net-prod" 192.168.103.200 \
|
||||
haproxy-02 10.10.10.51 80
|
||||
|
||||
# Map port 80 to both backends
|
||||
incus network load-balancer port add "${REMOTE}:net-prod" 192.168.103.200 \
|
||||
tcp 80 haproxy-01,haproxy-02
|
||||
|
||||
# Verify
|
||||
incus network load-balancer list "${REMOTE}:net-prod"
|
||||
```
|
||||
|
||||
Verify infrastructure:
|
||||
|
||||
```bash
|
||||
# Check via API
|
||||
curl -sSk -b "$COOKIE_JAR" \
|
||||
-H "X-CSRF-Token: ${CSRF}" \
|
||||
"${AETHER_URL}/haproxy/infrastructure/52"
|
||||
|
||||
# Check containers exist on cluster
|
||||
incus list oc-node-01: --format csv -c n | grep ffsdn-haproxy
|
||||
# ffsdn-haproxy-52-01
|
||||
# ffsdn-haproxy-52-02
|
||||
```
|
||||
|
||||
### Fix ACL rules for external access
|
||||
|
||||
Aether auto-generates per-instance ACLs that only allow ingress from
|
||||
`source: 192.168.103.200` (the VIP address). This works for traffic
|
||||
originating inside the OVN network — OVN hairpin-SNATs the source to
|
||||
the router IP (.200) when load-balancing back into the same network.
|
||||
However, external traffic (from outside OVN) preserves the original
|
||||
client IP, which doesn't match the ACL and gets **rejected**.
|
||||
|
||||
Add a broader ingress rule to each HAProxy container's ACL:
|
||||
|
||||
```bash
|
||||
REMOTE="oc-node-01"
|
||||
CLUSTER_ID=52
|
||||
|
||||
for n in 01 02; do
|
||||
ACL_NAME="${CLUSTER_ID}-ffsdn-haproxy-${CLUSTER_ID}-${n}-aether-acl"
|
||||
incus network acl rule add "${REMOTE}:${ACL_NAME}" ingress \
|
||||
action=allow protocol=tcp destination_port=80,443 \
|
||||
description="Allow external clients to reach HAProxy"
|
||||
done
|
||||
```
|
||||
|
||||
**Why is this needed?** OVN load balancers perform DNAT (rewriting the
|
||||
destination IP from the VIP to a backend), but for external-to-internal
|
||||
traffic they do NOT SNAT the source. The packet arrives at the HAProxy
|
||||
container with the external client's IP as source, which Aether's narrow
|
||||
ACL rule rejects. For internal traffic, OVN *does* hairpin-SNAT the source
|
||||
to .200 (to prevent asymmetric routing within the same logical switch),
|
||||
which is why it works from inside OVN without this fix.
|
||||
|
||||
**Note**: This ACL modification survives container restarts and
|
||||
reconfigurations. However, if you delete and redeploy the HAProxy
|
||||
infrastructure, Aether recreates the ACLs from scratch, so you'll
|
||||
need to re-apply this fix. Check future Aether versions — this may
|
||||
be fixed upstream.
|
||||
|
||||
## Step 4: Create service
|
||||
|
||||
Configure HAProxy to load-balance traffic across the nginx backends.
|
||||
|
||||
**Important: TCP vs HTTP mode** — For nginx backends, use **TCP mode**.
|
||||
HTTP mode generates `option httpchk` which sends `OPTIONS /` requests.
|
||||
Nginx returns `405 Not Allowed` for OPTIONS, causing HAProxy to mark
|
||||
all backends as DOWN. TCP mode uses simple TCP health checks (L4OK).
|
||||
|
||||
### Via Aether UI
|
||||
|
||||
1. In the **Services** section, click **+ Add Service**
|
||||
2. Fill in:
|
||||
- **Service Name**: `web-http`
|
||||
- **Description**: `Load balancer for nginx test backends`
|
||||
- **VIP**: `192.168.103.200 - Default`
|
||||
- **Listen Port**: `80`
|
||||
- **Mode**: `TCP (Layer 4)`
|
||||
- **Balance Method**: `Round Robin`
|
||||
- **Health Check**: enabled
|
||||
3. Add backend servers:
|
||||
- `nginx-lb-01` / `10.10.10.60` / port `80` / weight `100`
|
||||
- `nginx-lb-02` / `10.10.10.61` / port `80` / weight `100`
|
||||
- `nginx-lb-03` / `10.10.10.62` / port `80` / weight `100`
|
||||
4. Click **Create Service**
|
||||
|
||||
### Via session API
|
||||
|
||||
**Note**: The `vip_id` is the numeric VIP ID from the infrastructure
|
||||
deployment (visible in the API response or the Aether UI). For the
|
||||
default VIP created during deployment, retrieve it from
|
||||
`GET /haproxy/infrastructure/52` → `vips[0].id`.
|
||||
|
||||
```bash
|
||||
curl -sSk -b "$COOKIE_JAR" \
|
||||
-H "X-CSRF-Token: ${CSRF}" \
|
||||
-H "Referer: ${AETHER_URL}/haproxy" \
|
||||
-X POST "${AETHER_URL}/haproxy/service" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"vip_id": 42,
|
||||
"name": "web-http",
|
||||
"description": "HTTP load balancer for nginx test backends",
|
||||
"hostname": "",
|
||||
"listen_port": 80,
|
||||
"mode": "tcp",
|
||||
"balance_method": "roundrobin",
|
||||
"health_check_enabled": true,
|
||||
"session_persistence": false,
|
||||
"backends": [
|
||||
{"name": "nginx-lb-01", "target_ip": "10.10.10.60", "target_port": 80, "weight": 100},
|
||||
{"name": "nginx-lb-02", "target_ip": "10.10.10.61", "target_port": 80, "weight": 100},
|
||||
{"name": "nginx-lb-03", "target_ip": "10.10.10.62", "target_port": 80, "weight": 100}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
**Note**: After creating or editing a service, Aether may need to reload
|
||||
the HAProxy configuration on both instances. If the service doesn't respond
|
||||
immediately, click **Reload Config** in the Aether UI, or wait a few seconds
|
||||
for automatic propagation.
|
||||
|
||||
## Step 5: Test load balancing
|
||||
|
||||
### Internal connectivity (via HAProxy localhost)
|
||||
|
||||
Test HAProxy directly by curling localhost on each container. This
|
||||
bypasses the OVN load balancer and tests HAProxy's backend routing:
|
||||
|
||||
```bash
|
||||
# Direct test on HAProxy 01 (bypasses OVN LB)
|
||||
for i in $(seq 1 6); do
|
||||
incus exec oc-node-01:ffsdn-haproxy-52-01 -- \
|
||||
curl -s http://localhost/ | grep -o 'Backend: [^<]*'
|
||||
done
|
||||
# Expected: nginx-lb-01, nginx-lb-02, nginx-lb-03 in rotation
|
||||
|
||||
# Direct test on HAProxy 02
|
||||
incus exec oc-node-01:ffsdn-haproxy-52-02 -- curl -s http://localhost/
|
||||
# Expected: response from one of the backends
|
||||
```
|
||||
|
||||
**Note**: Curling the VIP (192.168.103.200) from within the OVN network
|
||||
does NOT work reliably. The manually-created OVN load balancer handles
|
||||
external-to-internal traffic but does not hairpin for internal-to-VIP
|
||||
requests from the same logical switch. Use localhost for internal tests.
|
||||
|
||||
### External connectivity (from outside OVN)
|
||||
|
||||
After applying the ACL fix in Step 3, the VIP is reachable from any
|
||||
machine on the LAN:
|
||||
|
||||
```bash
|
||||
# Single request from external machine
|
||||
curl -s http://192.168.103.200/
|
||||
# Expected: response from one of the backends
|
||||
|
||||
# Verify round-robin across all backends
|
||||
for i in $(seq 1 6); do
|
||||
curl -s http://192.168.103.200/ | grep -o '<h1>[^<]*</h1>'
|
||||
done
|
||||
# Expected: all three backends appear in rotation
|
||||
```
|
||||
|
||||
If external access fails with "Connection refused", the ACL fix was
|
||||
not applied. See Step 3 "Fix ACL rules for external access".
|
||||
|
||||
### Backend failover
|
||||
|
||||
```bash
|
||||
# Stop one backend
|
||||
incus stop oc-node-01:nginx-lb-03
|
||||
|
||||
# Wait for health check to detect (default: 5 seconds)
|
||||
sleep 10
|
||||
|
||||
# Verify traffic only goes to remaining backends (external)
|
||||
for i in $(seq 1 6); do
|
||||
curl -s http://192.168.103.200/ | grep -o '<h1>[^<]*</h1>'
|
||||
done
|
||||
# Expected: only nginx-lb-01 and nginx-lb-02
|
||||
|
||||
# Restart the backend
|
||||
incus start oc-node-01:nginx-lb-03
|
||||
|
||||
# Wait for health check to re-add
|
||||
sleep 10
|
||||
|
||||
# Verify all three are back in rotation
|
||||
for i in $(seq 1 6); do
|
||||
curl -s http://192.168.103.200/ | grep -o '<h1>[^<]*</h1>'
|
||||
done
|
||||
```
|
||||
|
||||
### HAProxy failover
|
||||
|
||||
```bash
|
||||
# Stop one HAProxy container
|
||||
incus stop oc-node-01:ffsdn-haproxy-52-02
|
||||
|
||||
# Wait for OVN to detect the dead backend
|
||||
sleep 15
|
||||
|
||||
# Some requests will succeed (routed to HAProxy 01), others may time out
|
||||
curl -s --connect-timeout 10 http://192.168.103.200/
|
||||
# Expected: response from one of the backends (via HAProxy 01)
|
||||
|
||||
# Restart
|
||||
incus start oc-node-01:ffsdn-haproxy-52-02
|
||||
```
|
||||
|
||||
**Note**: OVN load balancers do NOT have active health checking. When one
|
||||
HAProxy instance is stopped, the OVN LB continues to send ~50% of
|
||||
connections to the dead backend, causing those connections to time out.
|
||||
After 10-15 seconds OVN may detect the dead backend, but failover is
|
||||
not instant. For production use, consider Keepalived mode which provides
|
||||
true active/standby VIP failover with VRRP.
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Health and statistics
|
||||
|
||||
Each service has a Health & Stats view accessible via the Aether UI
|
||||
(**Services** > **Health & Stats** button) or the session API:
|
||||
|
||||
```bash
|
||||
curl -sSk -b "$COOKIE_JAR" \
|
||||
-H "X-CSRF-Token: ${CSRF}" \
|
||||
"${AETHER_URL}/haproxy/service/health-stats?cluster_id=52&service_name=web-http"
|
||||
```
|
||||
|
||||
### Statistics fields
|
||||
|
||||
**Backend server statistics** (from HAProxy health checks):
|
||||
|
||||
| Field | Description |
|
||||
|-------|-------------|
|
||||
| Status | `UP` (healthy) or `DOWN` (removed from rotation) |
|
||||
| Sessions | Current active / maximum seen |
|
||||
| Bytes In/Out | Total traffic to/from this backend |
|
||||
| Weight | Configured load distribution weight |
|
||||
| Last Check | Most recent health check result + response time (ms) |
|
||||
|
||||
**Frontend statistics** (combined for all services on the same port):
|
||||
|
||||
| Field | Description |
|
||||
|-------|-------------|
|
||||
| Current Sessions | Active client connections |
|
||||
| Max Sessions | Highest concurrent sessions seen |
|
||||
| Total Sessions | Cumulative since HAProxy started |
|
||||
| Session Rate | New sessions per second |
|
||||
| Bytes In/Out | Total client traffic |
|
||||
|
||||
**HAProxy node resources** (per container):
|
||||
|
||||
| Field | Description | Color thresholds |
|
||||
|-------|-------------|------------------|
|
||||
| CPU Usage | Current utilization % | Green <50%, Orange 50-79%, Red 80%+ |
|
||||
| Memory Usage | Current utilization | Green <50%, Orange 50-79%, Red 80%+ |
|
||||
|
||||
## Keepalived mode
|
||||
|
||||
For clusters without OVN networking, Aether supports Keepalived-based
|
||||
HAProxy deployment. Instead of an OVN load balancer, Keepalived provides
|
||||
VIP failover between the two HAProxy instances using VRRP.
|
||||
|
||||
Key differences:
|
||||
- **No OVN required** — works with bridge or macvlan networking
|
||||
- **VRRP failover** — active/standby instead of OVN's active/active LB
|
||||
- **Direct VIP** — the VIP floats between HAProxy instances directly
|
||||
|
||||
This mode is selected during infrastructure deployment when choosing a
|
||||
non-OVN network. The service configuration is identical regardless of
|
||||
the underlying HA mechanism.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Image build fails
|
||||
|
||||
```bash
|
||||
# Check build status
|
||||
curl -sSk -b "$COOKIE_JAR" \
|
||||
-H "X-CSRF-Token: ${CSRF}" \
|
||||
"${AETHER_URL}/haproxy/image/build-status"
|
||||
```
|
||||
|
||||
**Common causes:**
|
||||
- No internet access from the cluster (package downloads fail)
|
||||
- Cluster not reachable from Aether
|
||||
- Insufficient disk space for image build
|
||||
|
||||
**Fix:** Ensure the cluster nodes can reach the internet. Try building
|
||||
on a different cluster if available.
|
||||
|
||||
### Infrastructure deployment fails
|
||||
|
||||
**"No Image" in cluster dropdown:** The HAProxy image hasn't been pushed
|
||||
to this cluster. Build an image (Step 2), then push it to the cluster
|
||||
and set it as current.
|
||||
|
||||
**Containers not starting:** Check the Incus cluster has sufficient
|
||||
resources (CPU, memory) for two additional containers.
|
||||
|
||||
```bash
|
||||
# Check container state
|
||||
incus list oc-node-01: -c ns --format csv | grep ffsdn-haproxy
|
||||
```
|
||||
|
||||
### HAProxy container not starting after deploy
|
||||
|
||||
Aether may deploy both containers but only start one. This was observed
|
||||
with `ffsdn-haproxy-52-01` remaining in STOPPED state after deployment.
|
||||
|
||||
```bash
|
||||
# Check both containers
|
||||
incus list oc-node-01: -f compact -c nsN4 | grep ffsdn-haproxy
|
||||
|
||||
# Start the stopped container
|
||||
incus start oc-node-01:ffsdn-haproxy-52-01
|
||||
```
|
||||
|
||||
### Service not responding
|
||||
|
||||
**VIP not reachable from outside OVN ("Connection refused"):**
|
||||
Aether auto-generates per-instance ACL rules that only allow ingress
|
||||
from `source: 192.168.103.200` (the VIP address). External traffic
|
||||
preserves the original client IP as source, which the ACL rejects.
|
||||
Apply the ACL fix from Step 3:
|
||||
|
||||
```bash
|
||||
# Diagnose: check the ACL rules
|
||||
incus network acl show oc-node-01:52-ffsdn-haproxy-52-01-aether-acl
|
||||
|
||||
# If the only ingress rule has source: 192.168.103.200, add a broader rule:
|
||||
for n in 01 02; do
|
||||
incus network acl rule add "oc-node-01:52-ffsdn-haproxy-52-${n}-aether-acl" \
|
||||
ingress action=allow protocol=tcp destination_port=80,443 \
|
||||
description="Allow external clients to reach HAProxy"
|
||||
done
|
||||
|
||||
# Verify it works now:
|
||||
curl -s http://192.168.103.200/
|
||||
```
|
||||
|
||||
**Why internal access works but external doesn't:** OVN hairpin-SNATs
|
||||
traffic that originates inside the OVN network and gets load-balanced
|
||||
back into the same network (source becomes .200, matching the ACL).
|
||||
External traffic doesn't get SNAT'd, so the original client IP doesn't
|
||||
match the narrow ACL rule.
|
||||
|
||||
**Backends showing DOWN in health stats (HTTP mode):**
|
||||
When using **HTTP mode**, Aether generates `option httpchk` which sends
|
||||
an `OPTIONS /` request. Nginx returns `405 Not Allowed`, marking backends
|
||||
DOWN. **Use TCP mode** for nginx backends, or check the HAProxy stats:
|
||||
```bash
|
||||
# Check backend health from HAProxy container
|
||||
incus exec oc-node-01:ffsdn-haproxy-52-01 -- \
|
||||
curl -s http://localhost:8404/stats\;csv | grep backend_web
|
||||
|
||||
# Verify direct backend connectivity
|
||||
incus exec oc-node-01:ffsdn-haproxy-52-01 -- curl -s http://10.10.10.60/
|
||||
```
|
||||
|
||||
### Traffic not reaching backends
|
||||
|
||||
Even when the service is created and HAProxy is running, traffic may not
|
||||
flow if:
|
||||
|
||||
- **ACL rules blocking traffic** — Aether creates per-instance ACLs with
|
||||
default-deny. If the ACLs don't cover the traffic pattern, it gets
|
||||
rejected. Check with `incus network acl show <remote>:<acl-name>`.
|
||||
- **HAProxy config not reloaded** — click **Reload Config** in the Aether
|
||||
UI or wait for automatic propagation.
|
||||
- **Wrong backend port** — verify the backends are actually listening on
|
||||
the configured port (80 in our case).
|
||||
|
||||
Test connectivity directly from a HAProxy container:
|
||||
|
||||
```bash
|
||||
incus exec oc-node-01:ffsdn-haproxy-52-01 -- curl -s http://10.10.10.60/
|
||||
```
|
||||
|
||||
### Sticky sessions not working
|
||||
|
||||
Sticky sessions only work with **HTTP** and **HTTPS Termination** modes.
|
||||
They do NOT work with TCP or HTTPS Passthrough (HAProxy can't insert
|
||||
cookies without inspecting HTTP content). Verify your service mode.
|
||||
|
||||
### Advanced options not taking effect
|
||||
|
||||
Advanced options (compression, HSTS, rate limiting, access control) only
|
||||
apply to **HTTP** and **HTTPS Termination** modes. They have no effect on
|
||||
TCP or HTTPS Passthrough services, as those operate at L4 without
|
||||
inspecting HTTP content.
|
||||
|
||||
### Rate limiting blocking legitimate users
|
||||
|
||||
If users behind corporate NAT or VPN share a single IP, they collectively
|
||||
count against the per-IP rate limit. A web page loading 20 images generates
|
||||
21 requests. Start with higher limits (500+) and reduce based on monitoring.
|
||||
|
||||
### Session authentication issues
|
||||
|
||||
Aether's HAProxy endpoints use session authentication with CSRF protection.
|
||||
**curl-based session auth is unreliable** — use Playwright browser automation
|
||||
for reliable interaction (see `incusos/helpers/aether-browser`).
|
||||
|
||||
Common issues with curl-based auth:
|
||||
1. **CSRF 403**: The CSRF token binding requires proper browser cookie handling.
|
||||
Playwright handles this automatically.
|
||||
2. **Login page at `/`**: The login form is at root (`/`), NOT `/login`.
|
||||
`GET /login` returns 404. The form POSTs to `/login`.
|
||||
3. **Session expiry**: Re-authenticate by re-running the Playwright login.
|
||||
|
||||
### Cleanup
|
||||
|
||||
To completely remove HAProxy infrastructure:
|
||||
|
||||
```bash
|
||||
# Via Aether UI: HAProxy > select cluster > Delete Infrastructure
|
||||
|
||||
# Via session API (requires CSRF token from Playwright):
|
||||
curl -sSk -b "$COOKIE_JAR" \
|
||||
-H "X-CSRF-Token: ${CSRF}" \
|
||||
-H "Referer: ${AETHER_URL}/haproxy" \
|
||||
-X DELETE "${AETHER_URL}/haproxy/infrastructure/52"
|
||||
|
||||
# Remove the OVN load balancer (created manually, not cleaned by Aether)
|
||||
incus network load-balancer delete oc-node-01:net-prod 192.168.103.200
|
||||
|
||||
# Clean up test backends
|
||||
for name in nginx-lb-01 nginx-lb-02 nginx-lb-03; do
|
||||
incus delete "oc-node-01:${name}" --force
|
||||
done
|
||||
```
|
||||
|
||||
**Warning**: Deleting infrastructure stops all traffic immediately.
|
||||
It removes both HAProxy containers, all VIPs, all services, and
|
||||
associated firewall rules.
|
||||
Loading…
Reference in New Issue