26 KiB

Raw Blame History

HAProxy Load Balancing — Aether-Managed HA Proxy on OVN

Aether includes a full HAProxy management system that deploys a highly available load balancer pair on OVN networks. It handles image builds, HA pair deployment, VIP management, L7 service configuration, and health monitoring — all through its web UI and session-authenticated API.

This guide covers deploying HAProxy infrastructure on the lab cluster, creating test backends, configuring services, and verifying load balancing.

Architecture

flowchart TD
    client(["Client / LAN"])
    vip["VIP 192.168.103.200"]
    ovnlb("OVN Load Balancer")
    ha1["HAProxy 01<br/>10.10.10.50"]
    ha2["HAProxy 02<br/>10.10.10.51"]
    ng1["nginx-01 · .60"]
    ng2["nginx-02 · .61"]
    ng3["nginx-03 · .62"]

    client --> vip
    vip --> ovnlb
    ovnlb --> ha1 & ha2
    ha1 & ha2 --> ng1 & ng2 & ng3

    classDef external fill:#f5f5f5,color:#333,stroke:#999
    classDef network fill:#0072B2,color:#fff,stroke:#005a8e
    classDef lb fill:#E69F00,color:#fff,stroke:#b87d00
    classDef instance fill:#56B4E9,color:#fff,stroke:#3a8fbf

    class client external
    class vip,ovnlb network
    class ha1,ha2 lb
    class ng1,ng2,ng3 instance

OVN native LB vs HAProxy

Feature	OVN native LB	Aether HAProxy
Layer	L4 (TCP/UDP)	L4 + L7 (HTTP/HTTPS)
Health checks	None	HTTP, TCP, configurable interval
SSL termination	No	Yes (HTTPS Termination mode)
Sticky sessions	No	Yes (cookie-based)
Load balancing	Connection hash	Round robin, least conn, source hash, first available
SNI routing	No	Yes (multiple domains on same VIP:port)
Compression	No	Yes (gzip for text content)
Rate limiting	No	Yes (per-IP)
Management	`incus network load-balancer` CLI	Aether UI + session API
HA	Single OVN LB (distributed)	HA pair with OVN LB failover

Use OVN native LB for simple L4 distribution. Use HAProxy when you need health checks, L7 features, SSL, sticky sessions, or advanced traffic management.

Prerequisites

Incus cluster registered in Aether with OVN networking configured
OVN network with a physical UPLINK (for VIP addresses)
Available IP addresses on the OVN network for HAProxy instances
Available VIP address(es) from the UPLINK range
Internet access from the cluster (for HAProxy image builds)
incus CLI configured with cluster remote
Aether admin access (Full Admin or LB Admin role)

Resource requirements

Resource	Per HAProxy instance	Notes
CPU	2 cores (default)	HAProxy is very CPU-efficient
Memory	1 GiB (default)	Increase for many concurrent connections
Disk	Minimal	Container image, no persistent storage needed

Lab details

Setting	Value
Cluster	oc-lab-cluster (ID: 52)
OVN network	net-prod (10.10.10.0/24)
UPLINK	UPLINK (physical)
VIP range	192.168.103.200-210
HAProxy 01 IP	10.10.10.50
HAProxy 02 IP	10.10.10.51
Backend IPs	10.10.10.60, 10.10.10.61, 10.10.10.62
HAProxy containers	ffsdn-haproxy-52-01, ffsdn-haproxy-52-02
Aether URL	https://192.168.102.160:8443
Cluster remote	oc-node-01

Automated path

The incusos/deploy-haproxy script automates the entire deployment:

# Check prerequisites
deploy-haproxy --doctor

# Preview deployment
deploy-haproxy --deploy --dry-run

# Full deployment (backends + image + infrastructure + service)
deploy-haproxy --deploy

# Check health
deploy-haproxy --status

# Clean removal
deploy-haproxy --cleanup

The manual steps below are for reference, troubleshooting, and understanding what the script does.

Step 1: Deploy test backends

Create 3 nginx containers on net-prod to serve as load-balanced backends. Each gets a unique index page so we can verify traffic distribution.

Important: The backend IPs (10.10.10.60-62) must not already be in use on the OVN network. Check the Aether UI (HAProxy > select cluster) for used IPs before deploying, or use deploy-haproxy --doctor which checks for you.

REMOTE="oc-node-01"

for i in 60 61 62; do
    NAME="nginx-lb-$(printf '%02d' $((i - 59)))"
    IP="10.10.10.${i}"

    # Launch container on OVN network with static IP via device key
    incus launch images:debian/12 "${REMOTE}:${NAME}" \
        --network net-prod \
        --storage local \
        -d root,size=2GiB \
        -d eth0,ipv4.address="${IP}" \
        -c limits.cpu=1 \
        -c limits.memory=256MiB

    # Wait for container agent
    echo "Waiting for ${NAME}..."
    while ! incus exec "${REMOTE}:${NAME}" -- true 2>/dev/null; do
        sleep 2
    done

    # Install nginx + curl, create unique index page
    incus exec "${REMOTE}:${NAME}" -- bash -c "
        export DEBIAN_FRONTEND=noninteractive
        apt-get update -qq
        apt-get install -y -qq nginx curl 2>&1 | tail -1

        cat > /var/www/html/index.html << HTML
<html><body>
<h1>Backend: ${NAME}</h1>
<p>IP: ${IP}</p>
</body></html>
HTML

        systemctl enable nginx
        systemctl start nginx
    "

    echo "Backend ${NAME} deployed at ${IP}"
done

Verify all backends respond (curling from one backend to the others confirms OVN cross-container connectivity):

# Wait for static IPs to take effect
sleep 5

for i in 60 61 62; do
    incus exec "${REMOTE}:nginx-lb-01" -- \
        curl -s --connect-timeout 3 "http://10.10.10.${i}/" \
        | grep -o 'Backend: [^<]*'
done
# Expected:
# Backend: nginx-lb-01
# Backend: nginx-lb-02
# Backend: nginx-lb-03

If curl fails, the static IP may not have taken effect yet. Verify with incus exec "${REMOTE}:nginx-lb-01" -- ip addr show eth0.

Step 2: Build HAProxy image

Aether builds a base HAProxy container image and stores it internally. The image must be pushed to each target cluster before infrastructure can be deployed.

Note: The HAProxy management features are not in Aether's JWT API (/api/swagger.yaml). They use session-authenticated routes discovered from the UI JavaScript. The script uses session auth automatically.

Via Aether UI

Navigate to HAProxy in the main menu
In HAProxy Base Images, click Build New Image
Select Build on Cluster: oc-lab-cluster
Enter Version: 1.0.0
Click Build Image
Wait 5-10 minutes for the build to complete
Click Push to Cluster and select oc-lab-cluster
Click Set Current on the new image

Via session API

The build process uses session cookies + CSRF token authentication:

AETHER_URL="https://192.168.102.160:8443"
COOKIE_JAR="/tmp/aether-cookies.txt"

# Step 1: Get login page and extract CSRF token from hidden form field
# Note: the login form is at "/" (root), NOT "/login". GET /login returns 404.
CSRF=$(curl -sSk -c "$COOKIE_JAR" "${AETHER_URL}/" \
    | grep -o 'name="csrf_token" value="[^"]*"' \
    | sed 's/.*value="//;s/"//')

# Step 2: Login with CSRF token (form-urlencoded, NOT JSON)
curl -sSk -b "$COOKIE_JAR" -c "$COOKIE_JAR" \
    -X POST "${AETHER_URL}/login" \
    -d "csrf_token=${CSRF}&username=admin&password=${AETHER_ADMIN_PASSWORD}" \
    -L -o /dev/null

# Step 3: Get fresh CSRF token for API calls
CSRF=$(curl -sSk -b "$COOKIE_JAR" "${AETHER_URL}/haproxy" \
    | grep -o 'name="csrf_token" value="[^"]*"' \
    | sed 's/.*value="//;s/"//')

# Step 4: Build image
curl -sSk -b "$COOKIE_JAR" \
    -H "X-CSRF-Token: ${CSRF}" \
    -H "Referer: ${AETHER_URL}/haproxy" \
    -X POST "${AETHER_URL}/haproxy/image/build" \
    -H "Content-Type: application/json" \
    -d '{"cluster_id": 52, "version": "1.0.0"}'

# Step 5: Poll build status until complete
while true; do
    STATUS=$(curl -sSk -b "$COOKIE_JAR" \
        -H "X-CSRF-Token: ${CSRF}" \
        "${AETHER_URL}/haproxy/image/build-status" \
        | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('status','unknown'))")
    echo "Build status: $STATUS"
    [[ "$STATUS" == "completed" || "$STATUS" == "failed" ]] && break
    sleep 10
done

# Step 6: Get image ID and push to cluster
IMAGE_ID=$(curl -sSk -b "$COOKIE_JAR" \
    -H "X-CSRF-Token: ${CSRF}" \
    "${AETHER_URL}/haproxy/images" \
    | python3 -c "import sys,json; imgs=json.load(sys.stdin); print(imgs[-1]['id'] if imgs else '')")

curl -sSk -b "$COOKIE_JAR" \
    -H "X-CSRF-Token: ${CSRF}" \
    -H "Referer: ${AETHER_URL}/haproxy" \
    -X POST "${AETHER_URL}/haproxy/image/push" \
    -H "Content-Type: application/json" \
    -d "{\"image_id\": ${IMAGE_ID}, \"cluster_id\": 52}"

# Step 7: Set as current version
curl -sSk -b "$COOKIE_JAR" \
    -H "X-CSRF-Token: ${CSRF}" \
    -H "Referer: ${AETHER_URL}/haproxy" \
    -X POST "${AETHER_URL}/haproxy/image/set-current" \
    -H "Content-Type: application/json" \
    -d "{\"image_id\": ${IMAGE_ID}}"

Critical: The CSRF token comes from a hidden HTML form field (<input name="csrf_token" value="...">), NOT from the cookie value. The cookie contains the session ID; the form field contains the CSRF token.

Step 3: Deploy HAProxy infrastructure

Deploy the HA pair on the cluster, then create the OVN load balancer.

Via Aether UI

Navigate to HAProxy > select oc-lab-cluster from dropdown
Click Deploy HAProxy Infrastructure
Fill in:
- OVN Network: net-prod
- Load Balancer VIP: 192.168.103.200 (from UPLINK range dropdown)
- HAProxy 01 IP: 10.10.10.50
- HAProxy 02 IP: 10.10.10.51
- CPU Limit: 2
- Memory Limit: 1GB
Click Deploy

Via session API

curl -sSk -b "$COOKIE_JAR" \
    -H "X-CSRF-Token: ${CSRF}" \
    -H "Referer: ${AETHER_URL}/haproxy" \
    -X POST "${AETHER_URL}/haproxy/infrastructure/deploy" \
    -H "Content-Type: application/json" \
    -d '{
        "cluster_id": 52,
        "ovn_network": "net-prod",
        "lb_vip": "192.168.103.200",
        "haproxy_01_ip": "10.10.10.50",
        "haproxy_02_ip": "10.10.10.51",
        "cpu_limit": "2",
        "memory_limit": "1GB"
    }'

This creates:

Two containers: ffsdn-haproxy-52-01 and ffsdn-haproxy-52-02
Per-instance ACL rules allowing traffic from the VIP
A default VIP entry at 192.168.103.200

Important: Aether does NOT create the OVN load balancer automatically. You must create it manually after infrastructure deployment:

REMOTE="oc-node-01"

# Create OVN load balancer for the VIP
incus network load-balancer create "${REMOTE}:net-prod" 192.168.103.200 \
    --description "FFSDN HAProxy Load Balancer"

# Add both HAProxy instances as backends
incus network load-balancer backend add "${REMOTE}:net-prod" 192.168.103.200 \
    haproxy-01 10.10.10.50 80
incus network load-balancer backend add "${REMOTE}:net-prod" 192.168.103.200 \
    haproxy-02 10.10.10.51 80

# Map port 80 to both backends
incus network load-balancer port add "${REMOTE}:net-prod" 192.168.103.200 \
    tcp 80 haproxy-01,haproxy-02

# Verify
incus network load-balancer list "${REMOTE}:net-prod"

Verify infrastructure:

# Check via API
curl -sSk -b "$COOKIE_JAR" \
    -H "X-CSRF-Token: ${CSRF}" \
    "${AETHER_URL}/haproxy/infrastructure/52"

# Check containers exist on cluster
incus list oc-node-01: --format csv -c n | grep ffsdn-haproxy
# ffsdn-haproxy-52-01
# ffsdn-haproxy-52-02

Fix ACL rules for external access

Aether auto-generates per-instance ACLs that only allow ingress from source: 192.168.103.200 (the VIP address). This works for traffic originating inside the OVN network — OVN hairpin-SNATs the source to the router IP (.200) when load-balancing back into the same network. However, external traffic (from outside OVN) preserves the original client IP, which doesn't match the ACL and gets rejected.

Add a broader ingress rule to each HAProxy container's ACL:

REMOTE="oc-node-01"
CLUSTER_ID=52

for n in 01 02; do
    ACL_NAME="${CLUSTER_ID}-ffsdn-haproxy-${CLUSTER_ID}-${n}-aether-acl"
    incus network acl rule add "${REMOTE}:${ACL_NAME}" ingress \
        action=allow protocol=tcp destination_port=80,443 \
        description="Allow external clients to reach HAProxy"
done

Why is this needed? OVN load balancers perform DNAT (rewriting the destination IP from the VIP to a backend), but for external-to-internal traffic they do NOT SNAT the source. The packet arrives at the HAProxy container with the external client's IP as source, which Aether's narrow ACL rule rejects. For internal traffic, OVN does hairpin-SNAT the source to .200 (to prevent asymmetric routing within the same logical switch), which is why it works from inside OVN without this fix.

Note: This ACL modification survives container restarts and reconfigurations. However, if you delete and redeploy the HAProxy infrastructure, Aether recreates the ACLs from scratch, so you'll need to re-apply this fix. Check future Aether versions — this may be fixed upstream.

Step 4: Create service

Configure HAProxy to load-balance traffic across the nginx backends.

Important: TCP vs HTTP mode — For nginx backends, use TCP mode. HTTP mode generates option httpchk which sends OPTIONS / requests. Nginx returns 405 Not Allowed for OPTIONS, causing HAProxy to mark all backends as DOWN. TCP mode uses simple TCP health checks (L4OK).

Via Aether UI

In the Services section, click + Add Service
Fill in:
- Service Name: web-http
- Description: Load balancer for nginx test backends
- VIP: 192.168.103.200 - Default
- Listen Port: 80
- Mode: TCP (Layer 4)
- Balance Method: Round Robin
- Health Check: enabled
Add backend servers:
- nginx-lb-01 / 10.10.10.60 / port 80 / weight 100
- nginx-lb-02 / 10.10.10.61 / port 80 / weight 100
- nginx-lb-03 / 10.10.10.62 / port 80 / weight 100
Click Create Service

Via session API

Note: The vip_id is the numeric VIP ID from the infrastructure deployment (visible in the API response or the Aether UI). For the default VIP created during deployment, retrieve it from GET /haproxy/infrastructure/52 → vips[0].id.

curl -sSk -b "$COOKIE_JAR" \
    -H "X-CSRF-Token: ${CSRF}" \
    -H "Referer: ${AETHER_URL}/haproxy" \
    -X POST "${AETHER_URL}/haproxy/service" \
    -H "Content-Type: application/json" \
    -d '{
        "vip_id": 42,
        "name": "web-http",
        "description": "HTTP load balancer for nginx test backends",
        "hostname": "",
        "listen_port": 80,
        "mode": "tcp",
        "balance_method": "roundrobin",
        "health_check_enabled": true,
        "session_persistence": false,
        "backends": [
            {"name": "nginx-lb-01", "target_ip": "10.10.10.60", "target_port": 80, "weight": 100},
            {"name": "nginx-lb-02", "target_ip": "10.10.10.61", "target_port": 80, "weight": 100},
            {"name": "nginx-lb-03", "target_ip": "10.10.10.62", "target_port": 80, "weight": 100}
        ]
    }'

Note: After creating or editing a service, Aether may need to reload the HAProxy configuration on both instances. If the service doesn't respond immediately, click Reload Config in the Aether UI, or wait a few seconds for automatic propagation.

Step 5: Test load balancing

Internal connectivity (via HAProxy localhost)

Test HAProxy directly by curling localhost on each container. This bypasses the OVN load balancer and tests HAProxy's backend routing:

# Direct test on HAProxy 01 (bypasses OVN LB)
for i in $(seq 1 6); do
    incus exec oc-node-01:ffsdn-haproxy-52-01 -- \
        curl -s http://localhost/ | grep -o 'Backend: [^<]*'
done
# Expected: nginx-lb-01, nginx-lb-02, nginx-lb-03 in rotation

# Direct test on HAProxy 02
incus exec oc-node-01:ffsdn-haproxy-52-02 -- curl -s http://localhost/
# Expected: response from one of the backends

Note: Curling the VIP (192.168.103.200) from within the OVN network does NOT work reliably. The manually-created OVN load balancer handles external-to-internal traffic but does not hairpin for internal-to-VIP requests from the same logical switch. Use localhost for internal tests.

External connectivity (from outside OVN)

After applying the ACL fix in Step 3, the VIP is reachable from any machine on the LAN:

# Single request from external machine
curl -s http://192.168.103.200/
# Expected: response from one of the backends

# Verify round-robin across all backends
for i in $(seq 1 6); do
    curl -s http://192.168.103.200/ | grep -o '<h1>[^<]*</h1>'
done
# Expected: all three backends appear in rotation

If external access fails with "Connection refused", the ACL fix was not applied. See Step 3 "Fix ACL rules for external access".

Backend failover

# Stop one backend
incus stop oc-node-01:nginx-lb-03

# Wait for health check to detect (default: 5 seconds)
sleep 10

# Verify traffic only goes to remaining backends (external)
for i in $(seq 1 6); do
    curl -s http://192.168.103.200/ | grep -o '<h1>[^<]*</h1>'
done
# Expected: only nginx-lb-01 and nginx-lb-02

# Restart the backend
incus start oc-node-01:nginx-lb-03

# Wait for health check to re-add
sleep 10

# Verify all three are back in rotation
for i in $(seq 1 6); do
    curl -s http://192.168.103.200/ | grep -o '<h1>[^<]*</h1>'
done

HAProxy failover

# Stop one HAProxy container
incus stop oc-node-01:ffsdn-haproxy-52-02

# Wait for OVN to detect the dead backend
sleep 15

# Some requests will succeed (routed to HAProxy 01), others may time out
curl -s --connect-timeout 10 http://192.168.103.200/
# Expected: response from one of the backends (via HAProxy 01)

# Restart
incus start oc-node-01:ffsdn-haproxy-52-02

Note: OVN load balancers do NOT have active health checking. When one HAProxy instance is stopped, the OVN LB continues to send ~50% of connections to the dead backend, causing those connections to time out. After 10-15 seconds OVN may detect the dead backend, but failover is not instant. For production use, consider Keepalived mode which provides true active/standby VIP failover with VRRP.

Monitoring

Health and statistics

Each service has a Health & Stats view accessible via the Aether UI (Services > Health & Stats button) or the session API:

curl -sSk -b "$COOKIE_JAR" \
    -H "X-CSRF-Token: ${CSRF}" \
    "${AETHER_URL}/haproxy/service/health-stats?cluster_id=52&service_name=web-http"

Statistics fields

Backend server statistics (from HAProxy health checks):

Field	Description
Status	`UP` (healthy) or `DOWN` (removed from rotation)
Sessions	Current active / maximum seen
Bytes In/Out	Total traffic to/from this backend
Weight	Configured load distribution weight
Last Check	Most recent health check result + response time (ms)

Frontend statistics (combined for all services on the same port):

Field	Description
Current Sessions	Active client connections
Max Sessions	Highest concurrent sessions seen
Total Sessions	Cumulative since HAProxy started
Session Rate	New sessions per second
Bytes In/Out	Total client traffic

HAProxy node resources (per container):

Field	Description	Color thresholds
CPU Usage	Current utilization %	Green <50%, Orange 50-79%, Red 80%+
Memory Usage	Current utilization	Green <50%, Orange 50-79%, Red 80%+

Keepalived mode

For clusters without OVN networking, Aether supports Keepalived-based HAProxy deployment. Instead of an OVN load balancer, Keepalived provides VIP failover between the two HAProxy instances using VRRP.

Key differences:

No OVN required — works with bridge or macvlan networking
VRRP failover — active/standby instead of OVN's active/active LB
Direct VIP — the VIP floats between HAProxy instances directly

This mode is selected during infrastructure deployment when choosing a non-OVN network. The service configuration is identical regardless of the underlying HA mechanism.

Troubleshooting

Image build fails

# Check build status
curl -sSk -b "$COOKIE_JAR" \
    -H "X-CSRF-Token: ${CSRF}" \
    "${AETHER_URL}/haproxy/image/build-status"

Common causes:

No internet access from the cluster (package downloads fail)
Cluster not reachable from Aether
Insufficient disk space for image build

Fix: Ensure the cluster nodes can reach the internet. Try building on a different cluster if available.

Infrastructure deployment fails

"No Image" in cluster dropdown: The HAProxy image hasn't been pushed to this cluster. Build an image (Step 2), then push it to the cluster and set it as current.

Containers not starting: Check the Incus cluster has sufficient resources (CPU, memory) for two additional containers.

# Check container state
incus list oc-node-01: -c ns --format csv | grep ffsdn-haproxy

HAProxy container not starting after deploy

Aether may deploy both containers but only start one. This was observed with ffsdn-haproxy-52-01 remaining in STOPPED state after deployment.

# Check both containers
incus list oc-node-01: -f compact -c nsN4 | grep ffsdn-haproxy

# Start the stopped container
incus start oc-node-01:ffsdn-haproxy-52-01

Service not responding

VIP not reachable from outside OVN ("Connection refused"): Aether auto-generates per-instance ACL rules that only allow ingress from source: 192.168.103.200 (the VIP address). External traffic preserves the original client IP as source, which the ACL rejects. Apply the ACL fix from Step 3:

# Diagnose: check the ACL rules
incus network acl show oc-node-01:52-ffsdn-haproxy-52-01-aether-acl

# If the only ingress rule has source: 192.168.103.200, add a broader rule:
for n in 01 02; do
    incus network acl rule add "oc-node-01:52-ffsdn-haproxy-52-${n}-aether-acl" \
        ingress action=allow protocol=tcp destination_port=80,443 \
        description="Allow external clients to reach HAProxy"
done

# Verify it works now:
curl -s http://192.168.103.200/

Why internal access works but external doesn't: OVN hairpin-SNATs traffic that originates inside the OVN network and gets load-balanced back into the same network (source becomes .200, matching the ACL). External traffic doesn't get SNAT'd, so the original client IP doesn't match the narrow ACL rule.

Backends showing DOWN in health stats (HTTP mode): When using HTTP mode, Aether generates option httpchk which sends an OPTIONS / request. Nginx returns 405 Not Allowed, marking backends DOWN. Use TCP mode for nginx backends, or check the HAProxy stats:

# Check backend health from HAProxy container
incus exec oc-node-01:ffsdn-haproxy-52-01 -- \
    curl -s http://localhost:8404/stats\;csv | grep backend_web

# Verify direct backend connectivity
incus exec oc-node-01:ffsdn-haproxy-52-01 -- curl -s http://10.10.10.60/

Traffic not reaching backends

Even when the service is created and HAProxy is running, traffic may not flow if:

ACL rules blocking traffic — Aether creates per-instance ACLs with default-deny. If the ACLs don't cover the traffic pattern, it gets rejected. Check with incus network acl show <remote>:<acl-name>.
HAProxy config not reloaded — click Reload Config in the Aether UI or wait for automatic propagation.
Wrong backend port — verify the backends are actually listening on the configured port (80 in our case).

Test connectivity directly from a HAProxy container:

incus exec oc-node-01:ffsdn-haproxy-52-01 -- curl -s http://10.10.10.60/

Sticky sessions not working

Sticky sessions only work with HTTP and HTTPS Termination modes. They do NOT work with TCP or HTTPS Passthrough (HAProxy can't insert cookies without inspecting HTTP content). Verify your service mode.

Advanced options not taking effect

Advanced options (compression, HSTS, rate limiting, access control) only apply to HTTP and HTTPS Termination modes. They have no effect on TCP or HTTPS Passthrough services, as those operate at L4 without inspecting HTTP content.

Rate limiting blocking legitimate users

If users behind corporate NAT or VPN share a single IP, they collectively count against the per-IP rate limit. A web page loading 20 images generates 21 requests. Start with higher limits (500+) and reduce based on monitoring.

Session authentication issues

Aether's HAProxy endpoints use session authentication with CSRF protection. curl-based session auth is unreliable — use Playwright browser automation for reliable interaction (see incusos/helpers/aether-browser).

Common issues with curl-based auth:

CSRF 403: The CSRF token binding requires proper browser cookie handling. Playwright handles this automatically.
Login page at /: The login form is at root (/), NOT /login. GET /login returns 404. The form POSTs to /login.
Session expiry: Re-authenticate by re-running the Playwright login.

Cleanup

To completely remove HAProxy infrastructure:

# Via Aether UI: HAProxy > select cluster > Delete Infrastructure

# Via session API (requires CSRF token from Playwright):
curl -sSk -b "$COOKIE_JAR" \
    -H "X-CSRF-Token: ${CSRF}" \
    -H "Referer: ${AETHER_URL}/haproxy" \
    -X DELETE "${AETHER_URL}/haproxy/infrastructure/52"

# Remove the OVN load balancer (created manually, not cleaned by Aether)
incus network load-balancer delete oc-node-01:net-prod 192.168.103.200

# Clean up test backends
for name in nginx-lb-01 nginx-lb-02 nginx-lb-03; do
    incus delete "oc-node-01:${name}" --force
done

Warning: Deleting infrastructure stops all traffic immediately. It removes both HAProxy containers, all VIPs, all services, and associated firewall rules.

26 KiB Raw Blame History

HAProxy Load Balancing — Aether-Managed HA Proxy on OVN

Architecture

OVN native LB vs HAProxy

Prerequisites

Resource requirements

Lab details

Automated path

Step 1: Deploy test backends

Step 2: Build HAProxy image

Via Aether UI

Via session API

Step 3: Deploy HAProxy infrastructure

Via Aether UI

Via session API

Fix ACL rules for external access

Step 4: Create service

Via Aether UI

Via session API

Step 5: Test load balancing

Internal connectivity (via HAProxy localhost)

External connectivity (from outside OVN)

Backend failover

HAProxy failover

Monitoring

Health and statistics

Statistics fields

Keepalived mode

Troubleshooting

Image build fails

Infrastructure deployment fails

HAProxy container not starting after deploy

Service not responding

Traffic not reaching backends

Sticky sessions not working

Advanced options not taking effect

Rate limiting blocking legitimate users

Session authentication issues

Cleanup

26 KiB

Raw Blame History