781 lines
26 KiB
Markdown
781 lines
26 KiB
Markdown
# HAProxy Load Balancing — Aether-Managed HA Proxy on OVN
|
|
|
|
Aether includes a full HAProxy management system that deploys a highly
|
|
available load balancer pair on OVN networks. It handles image builds,
|
|
HA pair deployment, VIP management, L7 service configuration, and health
|
|
monitoring — all through its web UI and session-authenticated API.
|
|
|
|
This guide covers deploying HAProxy infrastructure on the lab cluster,
|
|
creating test backends, configuring services, and verifying load balancing.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
Client / LAN traffic
|
|
|
|
|
v
|
|
+---------------------------+
|
|
| UPLINK VIP Address |
|
|
| 192.168.103.200 |
|
|
+---------------------------+
|
|
|
|
|
v
|
|
+---------------------------+
|
|
| OVN Load Balancer |
|
|
| (distributes to HA |
|
|
| pair for failover) |
|
|
+---------------------------+
|
|
/ \
|
|
/ \
|
|
v v
|
|
+-----------------+ +-----------------+
|
|
| HAProxy 01 | | HAProxy 02 |
|
|
| 10.10.10.50 | | 10.10.10.51 |
|
|
| (container) | | (container) |
|
|
+-----------------+ +-----------------+
|
|
\ | /
|
|
\ | /
|
|
v v v
|
|
+----------+ +----------+ +----------+
|
|
| nginx-01 | | nginx-02 | | nginx-03 |
|
|
| .60 :80 | | .61 :80 | | .62 :80 |
|
|
+----------+ +----------+ +----------+
|
|
(backend servers on net-prod)
|
|
```
|
|
|
|
### OVN native LB vs HAProxy
|
|
|
|
| Feature | OVN native LB | Aether HAProxy |
|
|
|---------|---------------|----------------|
|
|
| Layer | L4 (TCP/UDP) | L4 + L7 (HTTP/HTTPS) |
|
|
| Health checks | None | HTTP, TCP, configurable interval |
|
|
| SSL termination | No | Yes (HTTPS Termination mode) |
|
|
| Sticky sessions | No | Yes (cookie-based) |
|
|
| Load balancing | Connection hash | Round robin, least conn, source hash, first available |
|
|
| SNI routing | No | Yes (multiple domains on same VIP:port) |
|
|
| Compression | No | Yes (gzip for text content) |
|
|
| Rate limiting | No | Yes (per-IP) |
|
|
| Management | `incus network load-balancer` CLI | Aether UI + session API |
|
|
| HA | Single OVN LB (distributed) | HA pair with OVN LB failover |
|
|
|
|
Use OVN native LB for simple L4 distribution. Use HAProxy when you need
|
|
health checks, L7 features, SSL, sticky sessions, or advanced traffic management.
|
|
|
|
## Prerequisites
|
|
|
|
- Incus cluster registered in Aether with OVN networking configured
|
|
- OVN network with a physical UPLINK (for VIP addresses)
|
|
- Available IP addresses on the OVN network for HAProxy instances
|
|
- Available VIP address(es) from the UPLINK range
|
|
- Internet access from the cluster (for HAProxy image builds)
|
|
- `incus` CLI configured with cluster remote
|
|
- Aether admin access (Full Admin or LB Admin role)
|
|
|
|
### Resource requirements
|
|
|
|
| Resource | Per HAProxy instance | Notes |
|
|
|----------|---------------------|-------|
|
|
| CPU | 2 cores (default) | HAProxy is very CPU-efficient |
|
|
| Memory | 1 GiB (default) | Increase for many concurrent connections |
|
|
| Disk | Minimal | Container image, no persistent storage needed |
|
|
|
|
## Lab details
|
|
|
|
| Setting | Value |
|
|
|---------|-------|
|
|
| Cluster | oc-lab-cluster (ID: 52) |
|
|
| OVN network | net-prod (10.10.10.0/24) |
|
|
| UPLINK | UPLINK (physical) |
|
|
| VIP range | 192.168.103.200-210 |
|
|
| HAProxy 01 IP | 10.10.10.50 |
|
|
| HAProxy 02 IP | 10.10.10.51 |
|
|
| Backend IPs | 10.10.10.60, 10.10.10.61, 10.10.10.62 |
|
|
| HAProxy containers | ffsdn-haproxy-52-01, ffsdn-haproxy-52-02 |
|
|
| Aether URL | https://192.168.102.160:8443 |
|
|
| Cluster remote | oc-node-01 |
|
|
|
|
## Automated path
|
|
|
|
The `incusos/deploy-haproxy` script automates the entire deployment:
|
|
|
|
```bash
|
|
# Check prerequisites
|
|
deploy-haproxy --doctor
|
|
|
|
# Preview deployment
|
|
deploy-haproxy --deploy --dry-run
|
|
|
|
# Full deployment (backends + image + infrastructure + service)
|
|
deploy-haproxy --deploy
|
|
|
|
# Check health
|
|
deploy-haproxy --status
|
|
|
|
# Clean removal
|
|
deploy-haproxy --cleanup
|
|
```
|
|
|
|
The manual steps below are for reference, troubleshooting, and understanding
|
|
what the script does.
|
|
|
|
## Step 1: Deploy test backends
|
|
|
|
Create 3 nginx containers on net-prod to serve as load-balanced backends.
|
|
Each gets a unique index page so we can verify traffic distribution.
|
|
|
|
**Important**: The backend IPs (10.10.10.60-62) must not already be in use
|
|
on the OVN network. Check the Aether UI (HAProxy > select cluster) for
|
|
used IPs before deploying, or use `deploy-haproxy --doctor` which checks
|
|
for you.
|
|
|
|
```bash
|
|
REMOTE="oc-node-01"
|
|
|
|
for i in 60 61 62; do
|
|
NAME="nginx-lb-$(printf '%02d' $((i - 59)))"
|
|
IP="10.10.10.${i}"
|
|
|
|
# Launch container on OVN network with static IP via device key
|
|
incus launch images:debian/12 "${REMOTE}:${NAME}" \
|
|
--network net-prod \
|
|
--storage local \
|
|
-d root,size=2GiB \
|
|
-d eth0,ipv4.address="${IP}" \
|
|
-c limits.cpu=1 \
|
|
-c limits.memory=256MiB
|
|
|
|
# Wait for container agent
|
|
echo "Waiting for ${NAME}..."
|
|
while ! incus exec "${REMOTE}:${NAME}" -- true 2>/dev/null; do
|
|
sleep 2
|
|
done
|
|
|
|
# Install nginx + curl, create unique index page
|
|
incus exec "${REMOTE}:${NAME}" -- bash -c "
|
|
export DEBIAN_FRONTEND=noninteractive
|
|
apt-get update -qq
|
|
apt-get install -y -qq nginx curl 2>&1 | tail -1
|
|
|
|
cat > /var/www/html/index.html << HTML
|
|
<html><body>
|
|
<h1>Backend: ${NAME}</h1>
|
|
<p>IP: ${IP}</p>
|
|
</body></html>
|
|
HTML
|
|
|
|
systemctl enable nginx
|
|
systemctl start nginx
|
|
"
|
|
|
|
echo "Backend ${NAME} deployed at ${IP}"
|
|
done
|
|
```
|
|
|
|
Verify all backends respond (curling from one backend to the others
|
|
confirms OVN cross-container connectivity):
|
|
|
|
```bash
|
|
# Wait for static IPs to take effect
|
|
sleep 5
|
|
|
|
for i in 60 61 62; do
|
|
incus exec "${REMOTE}:nginx-lb-01" -- \
|
|
curl -s --connect-timeout 3 "http://10.10.10.${i}/" \
|
|
| grep -o 'Backend: [^<]*'
|
|
done
|
|
# Expected:
|
|
# Backend: nginx-lb-01
|
|
# Backend: nginx-lb-02
|
|
# Backend: nginx-lb-03
|
|
```
|
|
|
|
If curl fails, the static IP may not have taken effect yet. Verify with
|
|
`incus exec "${REMOTE}:nginx-lb-01" -- ip addr show eth0`.
|
|
|
|
## Step 2: Build HAProxy image
|
|
|
|
Aether builds a base HAProxy container image and stores it internally.
|
|
The image must be pushed to each target cluster before infrastructure
|
|
can be deployed.
|
|
|
|
**Note**: The HAProxy management features are not in Aether's JWT API
|
|
(`/api/swagger.yaml`). They use session-authenticated routes discovered
|
|
from the UI JavaScript. The script uses session auth automatically.
|
|
|
|
### Via Aether UI
|
|
|
|
1. Navigate to **HAProxy** in the main menu
|
|
2. In **HAProxy Base Images**, click **Build New Image**
|
|
3. Select **Build on Cluster**: `oc-lab-cluster`
|
|
4. Enter **Version**: `1.0.0`
|
|
5. Click **Build Image**
|
|
6. Wait 5-10 minutes for the build to complete
|
|
7. Click **Push to Cluster** and select `oc-lab-cluster`
|
|
8. Click **Set Current** on the new image
|
|
|
|
### Via session API
|
|
|
|
The build process uses session cookies + CSRF token authentication:
|
|
|
|
```bash
|
|
AETHER_URL="https://192.168.102.160:8443"
|
|
COOKIE_JAR="/tmp/aether-cookies.txt"
|
|
|
|
# Step 1: Get login page and extract CSRF token from hidden form field
|
|
# Note: the login form is at "/" (root), NOT "/login". GET /login returns 404.
|
|
CSRF=$(curl -sSk -c "$COOKIE_JAR" "${AETHER_URL}/" \
|
|
| grep -o 'name="csrf_token" value="[^"]*"' \
|
|
| sed 's/.*value="//;s/"//')
|
|
|
|
# Step 2: Login with CSRF token (form-urlencoded, NOT JSON)
|
|
curl -sSk -b "$COOKIE_JAR" -c "$COOKIE_JAR" \
|
|
-X POST "${AETHER_URL}/login" \
|
|
-d "csrf_token=${CSRF}&username=admin&password=${AETHER_ADMIN_PASSWORD}" \
|
|
-L -o /dev/null
|
|
|
|
# Step 3: Get fresh CSRF token for API calls
|
|
CSRF=$(curl -sSk -b "$COOKIE_JAR" "${AETHER_URL}/haproxy" \
|
|
| grep -o 'name="csrf_token" value="[^"]*"' \
|
|
| sed 's/.*value="//;s/"//')
|
|
|
|
# Step 4: Build image
|
|
curl -sSk -b "$COOKIE_JAR" \
|
|
-H "X-CSRF-Token: ${CSRF}" \
|
|
-H "Referer: ${AETHER_URL}/haproxy" \
|
|
-X POST "${AETHER_URL}/haproxy/image/build" \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"cluster_id": 52, "version": "1.0.0"}'
|
|
|
|
# Step 5: Poll build status until complete
|
|
while true; do
|
|
STATUS=$(curl -sSk -b "$COOKIE_JAR" \
|
|
-H "X-CSRF-Token: ${CSRF}" \
|
|
"${AETHER_URL}/haproxy/image/build-status" \
|
|
| python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('status','unknown'))")
|
|
echo "Build status: $STATUS"
|
|
[[ "$STATUS" == "completed" || "$STATUS" == "failed" ]] && break
|
|
sleep 10
|
|
done
|
|
|
|
# Step 6: Get image ID and push to cluster
|
|
IMAGE_ID=$(curl -sSk -b "$COOKIE_JAR" \
|
|
-H "X-CSRF-Token: ${CSRF}" \
|
|
"${AETHER_URL}/haproxy/images" \
|
|
| python3 -c "import sys,json; imgs=json.load(sys.stdin); print(imgs[-1]['id'] if imgs else '')")
|
|
|
|
curl -sSk -b "$COOKIE_JAR" \
|
|
-H "X-CSRF-Token: ${CSRF}" \
|
|
-H "Referer: ${AETHER_URL}/haproxy" \
|
|
-X POST "${AETHER_URL}/haproxy/image/push" \
|
|
-H "Content-Type: application/json" \
|
|
-d "{\"image_id\": ${IMAGE_ID}, \"cluster_id\": 52}"
|
|
|
|
# Step 7: Set as current version
|
|
curl -sSk -b "$COOKIE_JAR" \
|
|
-H "X-CSRF-Token: ${CSRF}" \
|
|
-H "Referer: ${AETHER_URL}/haproxy" \
|
|
-X POST "${AETHER_URL}/haproxy/image/set-current" \
|
|
-H "Content-Type: application/json" \
|
|
-d "{\"image_id\": ${IMAGE_ID}}"
|
|
```
|
|
|
|
**Critical**: The CSRF token comes from a hidden HTML form field
|
|
(`<input name="csrf_token" value="...">`), NOT from the cookie value.
|
|
The cookie contains the session ID; the form field contains the CSRF token.
|
|
|
|
## Step 3: Deploy HAProxy infrastructure
|
|
|
|
Deploy the HA pair on the cluster, then create the OVN load balancer.
|
|
|
|
### Via Aether UI
|
|
|
|
1. Navigate to **HAProxy** > select **oc-lab-cluster** from dropdown
|
|
2. Click **Deploy HAProxy Infrastructure**
|
|
3. Fill in:
|
|
- **OVN Network**: `net-prod`
|
|
- **Load Balancer VIP**: `192.168.103.200` (from UPLINK range dropdown)
|
|
- **HAProxy 01 IP**: `10.10.10.50`
|
|
- **HAProxy 02 IP**: `10.10.10.51`
|
|
- **CPU Limit**: `2`
|
|
- **Memory Limit**: `1GB`
|
|
4. Click **Deploy**
|
|
|
|
### Via session API
|
|
|
|
```bash
|
|
curl -sSk -b "$COOKIE_JAR" \
|
|
-H "X-CSRF-Token: ${CSRF}" \
|
|
-H "Referer: ${AETHER_URL}/haproxy" \
|
|
-X POST "${AETHER_URL}/haproxy/infrastructure/deploy" \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"cluster_id": 52,
|
|
"ovn_network": "net-prod",
|
|
"lb_vip": "192.168.103.200",
|
|
"haproxy_01_ip": "10.10.10.50",
|
|
"haproxy_02_ip": "10.10.10.51",
|
|
"cpu_limit": "2",
|
|
"memory_limit": "1GB"
|
|
}'
|
|
```
|
|
|
|
This creates:
|
|
- Two containers: `ffsdn-haproxy-52-01` and `ffsdn-haproxy-52-02`
|
|
- Per-instance ACL rules allowing traffic from the VIP
|
|
- A default VIP entry at 192.168.103.200
|
|
|
|
**Important**: Aether does NOT create the OVN load balancer automatically.
|
|
You must create it manually after infrastructure deployment:
|
|
|
|
```bash
|
|
REMOTE="oc-node-01"
|
|
|
|
# Create OVN load balancer for the VIP
|
|
incus network load-balancer create "${REMOTE}:net-prod" 192.168.103.200 \
|
|
--description "FFSDN HAProxy Load Balancer"
|
|
|
|
# Add both HAProxy instances as backends
|
|
incus network load-balancer backend add "${REMOTE}:net-prod" 192.168.103.200 \
|
|
haproxy-01 10.10.10.50 80
|
|
incus network load-balancer backend add "${REMOTE}:net-prod" 192.168.103.200 \
|
|
haproxy-02 10.10.10.51 80
|
|
|
|
# Map port 80 to both backends
|
|
incus network load-balancer port add "${REMOTE}:net-prod" 192.168.103.200 \
|
|
tcp 80 haproxy-01,haproxy-02
|
|
|
|
# Verify
|
|
incus network load-balancer list "${REMOTE}:net-prod"
|
|
```
|
|
|
|
Verify infrastructure:
|
|
|
|
```bash
|
|
# Check via API
|
|
curl -sSk -b "$COOKIE_JAR" \
|
|
-H "X-CSRF-Token: ${CSRF}" \
|
|
"${AETHER_URL}/haproxy/infrastructure/52"
|
|
|
|
# Check containers exist on cluster
|
|
incus list oc-node-01: --format csv -c n | grep ffsdn-haproxy
|
|
# ffsdn-haproxy-52-01
|
|
# ffsdn-haproxy-52-02
|
|
```
|
|
|
|
### Fix ACL rules for external access
|
|
|
|
Aether auto-generates per-instance ACLs that only allow ingress from
|
|
`source: 192.168.103.200` (the VIP address). This works for traffic
|
|
originating inside the OVN network — OVN hairpin-SNATs the source to
|
|
the router IP (.200) when load-balancing back into the same network.
|
|
However, external traffic (from outside OVN) preserves the original
|
|
client IP, which doesn't match the ACL and gets **rejected**.
|
|
|
|
Add a broader ingress rule to each HAProxy container's ACL:
|
|
|
|
```bash
|
|
REMOTE="oc-node-01"
|
|
CLUSTER_ID=52
|
|
|
|
for n in 01 02; do
|
|
ACL_NAME="${CLUSTER_ID}-ffsdn-haproxy-${CLUSTER_ID}-${n}-aether-acl"
|
|
incus network acl rule add "${REMOTE}:${ACL_NAME}" ingress \
|
|
action=allow protocol=tcp destination_port=80,443 \
|
|
description="Allow external clients to reach HAProxy"
|
|
done
|
|
```
|
|
|
|
**Why is this needed?** OVN load balancers perform DNAT (rewriting the
|
|
destination IP from the VIP to a backend), but for external-to-internal
|
|
traffic they do NOT SNAT the source. The packet arrives at the HAProxy
|
|
container with the external client's IP as source, which Aether's narrow
|
|
ACL rule rejects. For internal traffic, OVN *does* hairpin-SNAT the source
|
|
to .200 (to prevent asymmetric routing within the same logical switch),
|
|
which is why it works from inside OVN without this fix.
|
|
|
|
**Note**: This ACL modification survives container restarts and
|
|
reconfigurations. However, if you delete and redeploy the HAProxy
|
|
infrastructure, Aether recreates the ACLs from scratch, so you'll
|
|
need to re-apply this fix. Check future Aether versions — this may
|
|
be fixed upstream.
|
|
|
|
## Step 4: Create service
|
|
|
|
Configure HAProxy to load-balance traffic across the nginx backends.
|
|
|
|
**Important: TCP vs HTTP mode** — For nginx backends, use **TCP mode**.
|
|
HTTP mode generates `option httpchk` which sends `OPTIONS /` requests.
|
|
Nginx returns `405 Not Allowed` for OPTIONS, causing HAProxy to mark
|
|
all backends as DOWN. TCP mode uses simple TCP health checks (L4OK).
|
|
|
|
### Via Aether UI
|
|
|
|
1. In the **Services** section, click **+ Add Service**
|
|
2. Fill in:
|
|
- **Service Name**: `web-http`
|
|
- **Description**: `Load balancer for nginx test backends`
|
|
- **VIP**: `192.168.103.200 - Default`
|
|
- **Listen Port**: `80`
|
|
- **Mode**: `TCP (Layer 4)`
|
|
- **Balance Method**: `Round Robin`
|
|
- **Health Check**: enabled
|
|
3. Add backend servers:
|
|
- `nginx-lb-01` / `10.10.10.60` / port `80` / weight `100`
|
|
- `nginx-lb-02` / `10.10.10.61` / port `80` / weight `100`
|
|
- `nginx-lb-03` / `10.10.10.62` / port `80` / weight `100`
|
|
4. Click **Create Service**
|
|
|
|
### Via session API
|
|
|
|
**Note**: The `vip_id` is the numeric VIP ID from the infrastructure
|
|
deployment (visible in the API response or the Aether UI). For the
|
|
default VIP created during deployment, retrieve it from
|
|
`GET /haproxy/infrastructure/52` → `vips[0].id`.
|
|
|
|
```bash
|
|
curl -sSk -b "$COOKIE_JAR" \
|
|
-H "X-CSRF-Token: ${CSRF}" \
|
|
-H "Referer: ${AETHER_URL}/haproxy" \
|
|
-X POST "${AETHER_URL}/haproxy/service" \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"vip_id": 42,
|
|
"name": "web-http",
|
|
"description": "HTTP load balancer for nginx test backends",
|
|
"hostname": "",
|
|
"listen_port": 80,
|
|
"mode": "tcp",
|
|
"balance_method": "roundrobin",
|
|
"health_check_enabled": true,
|
|
"session_persistence": false,
|
|
"backends": [
|
|
{"name": "nginx-lb-01", "target_ip": "10.10.10.60", "target_port": 80, "weight": 100},
|
|
{"name": "nginx-lb-02", "target_ip": "10.10.10.61", "target_port": 80, "weight": 100},
|
|
{"name": "nginx-lb-03", "target_ip": "10.10.10.62", "target_port": 80, "weight": 100}
|
|
]
|
|
}'
|
|
```
|
|
|
|
**Note**: After creating or editing a service, Aether may need to reload
|
|
the HAProxy configuration on both instances. If the service doesn't respond
|
|
immediately, click **Reload Config** in the Aether UI, or wait a few seconds
|
|
for automatic propagation.
|
|
|
|
## Step 5: Test load balancing
|
|
|
|
### Internal connectivity (via HAProxy localhost)
|
|
|
|
Test HAProxy directly by curling localhost on each container. This
|
|
bypasses the OVN load balancer and tests HAProxy's backend routing:
|
|
|
|
```bash
|
|
# Direct test on HAProxy 01 (bypasses OVN LB)
|
|
for i in $(seq 1 6); do
|
|
incus exec oc-node-01:ffsdn-haproxy-52-01 -- \
|
|
curl -s http://localhost/ | grep -o 'Backend: [^<]*'
|
|
done
|
|
# Expected: nginx-lb-01, nginx-lb-02, nginx-lb-03 in rotation
|
|
|
|
# Direct test on HAProxy 02
|
|
incus exec oc-node-01:ffsdn-haproxy-52-02 -- curl -s http://localhost/
|
|
# Expected: response from one of the backends
|
|
```
|
|
|
|
**Note**: Curling the VIP (192.168.103.200) from within the OVN network
|
|
does NOT work reliably. The manually-created OVN load balancer handles
|
|
external-to-internal traffic but does not hairpin for internal-to-VIP
|
|
requests from the same logical switch. Use localhost for internal tests.
|
|
|
|
### External connectivity (from outside OVN)
|
|
|
|
After applying the ACL fix in Step 3, the VIP is reachable from any
|
|
machine on the LAN:
|
|
|
|
```bash
|
|
# Single request from external machine
|
|
curl -s http://192.168.103.200/
|
|
# Expected: response from one of the backends
|
|
|
|
# Verify round-robin across all backends
|
|
for i in $(seq 1 6); do
|
|
curl -s http://192.168.103.200/ | grep -o '<h1>[^<]*</h1>'
|
|
done
|
|
# Expected: all three backends appear in rotation
|
|
```
|
|
|
|
If external access fails with "Connection refused", the ACL fix was
|
|
not applied. See Step 3 "Fix ACL rules for external access".
|
|
|
|
### Backend failover
|
|
|
|
```bash
|
|
# Stop one backend
|
|
incus stop oc-node-01:nginx-lb-03
|
|
|
|
# Wait for health check to detect (default: 5 seconds)
|
|
sleep 10
|
|
|
|
# Verify traffic only goes to remaining backends (external)
|
|
for i in $(seq 1 6); do
|
|
curl -s http://192.168.103.200/ | grep -o '<h1>[^<]*</h1>'
|
|
done
|
|
# Expected: only nginx-lb-01 and nginx-lb-02
|
|
|
|
# Restart the backend
|
|
incus start oc-node-01:nginx-lb-03
|
|
|
|
# Wait for health check to re-add
|
|
sleep 10
|
|
|
|
# Verify all three are back in rotation
|
|
for i in $(seq 1 6); do
|
|
curl -s http://192.168.103.200/ | grep -o '<h1>[^<]*</h1>'
|
|
done
|
|
```
|
|
|
|
### HAProxy failover
|
|
|
|
```bash
|
|
# Stop one HAProxy container
|
|
incus stop oc-node-01:ffsdn-haproxy-52-02
|
|
|
|
# Wait for OVN to detect the dead backend
|
|
sleep 15
|
|
|
|
# Some requests will succeed (routed to HAProxy 01), others may time out
|
|
curl -s --connect-timeout 10 http://192.168.103.200/
|
|
# Expected: response from one of the backends (via HAProxy 01)
|
|
|
|
# Restart
|
|
incus start oc-node-01:ffsdn-haproxy-52-02
|
|
```
|
|
|
|
**Note**: OVN load balancers do NOT have active health checking. When one
|
|
HAProxy instance is stopped, the OVN LB continues to send ~50% of
|
|
connections to the dead backend, causing those connections to time out.
|
|
After 10-15 seconds OVN may detect the dead backend, but failover is
|
|
not instant. For production use, consider Keepalived mode which provides
|
|
true active/standby VIP failover with VRRP.
|
|
|
|
## Monitoring
|
|
|
|
### Health and statistics
|
|
|
|
Each service has a Health & Stats view accessible via the Aether UI
|
|
(**Services** > **Health & Stats** button) or the session API:
|
|
|
|
```bash
|
|
curl -sSk -b "$COOKIE_JAR" \
|
|
-H "X-CSRF-Token: ${CSRF}" \
|
|
"${AETHER_URL}/haproxy/service/health-stats?cluster_id=52&service_name=web-http"
|
|
```
|
|
|
|
### Statistics fields
|
|
|
|
**Backend server statistics** (from HAProxy health checks):
|
|
|
|
| Field | Description |
|
|
|-------|-------------|
|
|
| Status | `UP` (healthy) or `DOWN` (removed from rotation) |
|
|
| Sessions | Current active / maximum seen |
|
|
| Bytes In/Out | Total traffic to/from this backend |
|
|
| Weight | Configured load distribution weight |
|
|
| Last Check | Most recent health check result + response time (ms) |
|
|
|
|
**Frontend statistics** (combined for all services on the same port):
|
|
|
|
| Field | Description |
|
|
|-------|-------------|
|
|
| Current Sessions | Active client connections |
|
|
| Max Sessions | Highest concurrent sessions seen |
|
|
| Total Sessions | Cumulative since HAProxy started |
|
|
| Session Rate | New sessions per second |
|
|
| Bytes In/Out | Total client traffic |
|
|
|
|
**HAProxy node resources** (per container):
|
|
|
|
| Field | Description | Color thresholds |
|
|
|-------|-------------|------------------|
|
|
| CPU Usage | Current utilization % | Green <50%, Orange 50-79%, Red 80%+ |
|
|
| Memory Usage | Current utilization | Green <50%, Orange 50-79%, Red 80%+ |
|
|
|
|
## Keepalived mode
|
|
|
|
For clusters without OVN networking, Aether supports Keepalived-based
|
|
HAProxy deployment. Instead of an OVN load balancer, Keepalived provides
|
|
VIP failover between the two HAProxy instances using VRRP.
|
|
|
|
Key differences:
|
|
- **No OVN required** — works with bridge or macvlan networking
|
|
- **VRRP failover** — active/standby instead of OVN's active/active LB
|
|
- **Direct VIP** — the VIP floats between HAProxy instances directly
|
|
|
|
This mode is selected during infrastructure deployment when choosing a
|
|
non-OVN network. The service configuration is identical regardless of
|
|
the underlying HA mechanism.
|
|
|
|
## Troubleshooting
|
|
|
|
### Image build fails
|
|
|
|
```bash
|
|
# Check build status
|
|
curl -sSk -b "$COOKIE_JAR" \
|
|
-H "X-CSRF-Token: ${CSRF}" \
|
|
"${AETHER_URL}/haproxy/image/build-status"
|
|
```
|
|
|
|
**Common causes:**
|
|
- No internet access from the cluster (package downloads fail)
|
|
- Cluster not reachable from Aether
|
|
- Insufficient disk space for image build
|
|
|
|
**Fix:** Ensure the cluster nodes can reach the internet. Try building
|
|
on a different cluster if available.
|
|
|
|
### Infrastructure deployment fails
|
|
|
|
**"No Image" in cluster dropdown:** The HAProxy image hasn't been pushed
|
|
to this cluster. Build an image (Step 2), then push it to the cluster
|
|
and set it as current.
|
|
|
|
**Containers not starting:** Check the Incus cluster has sufficient
|
|
resources (CPU, memory) for two additional containers.
|
|
|
|
```bash
|
|
# Check container state
|
|
incus list oc-node-01: -c ns --format csv | grep ffsdn-haproxy
|
|
```
|
|
|
|
### HAProxy container not starting after deploy
|
|
|
|
Aether may deploy both containers but only start one. This was observed
|
|
with `ffsdn-haproxy-52-01` remaining in STOPPED state after deployment.
|
|
|
|
```bash
|
|
# Check both containers
|
|
incus list oc-node-01: -f compact -c nsN4 | grep ffsdn-haproxy
|
|
|
|
# Start the stopped container
|
|
incus start oc-node-01:ffsdn-haproxy-52-01
|
|
```
|
|
|
|
### Service not responding
|
|
|
|
**VIP not reachable from outside OVN ("Connection refused"):**
|
|
Aether auto-generates per-instance ACL rules that only allow ingress
|
|
from `source: 192.168.103.200` (the VIP address). External traffic
|
|
preserves the original client IP as source, which the ACL rejects.
|
|
Apply the ACL fix from Step 3:
|
|
|
|
```bash
|
|
# Diagnose: check the ACL rules
|
|
incus network acl show oc-node-01:52-ffsdn-haproxy-52-01-aether-acl
|
|
|
|
# If the only ingress rule has source: 192.168.103.200, add a broader rule:
|
|
for n in 01 02; do
|
|
incus network acl rule add "oc-node-01:52-ffsdn-haproxy-52-${n}-aether-acl" \
|
|
ingress action=allow protocol=tcp destination_port=80,443 \
|
|
description="Allow external clients to reach HAProxy"
|
|
done
|
|
|
|
# Verify it works now:
|
|
curl -s http://192.168.103.200/
|
|
```
|
|
|
|
**Why internal access works but external doesn't:** OVN hairpin-SNATs
|
|
traffic that originates inside the OVN network and gets load-balanced
|
|
back into the same network (source becomes .200, matching the ACL).
|
|
External traffic doesn't get SNAT'd, so the original client IP doesn't
|
|
match the narrow ACL rule.
|
|
|
|
**Backends showing DOWN in health stats (HTTP mode):**
|
|
When using **HTTP mode**, Aether generates `option httpchk` which sends
|
|
an `OPTIONS /` request. Nginx returns `405 Not Allowed`, marking backends
|
|
DOWN. **Use TCP mode** for nginx backends, or check the HAProxy stats:
|
|
```bash
|
|
# Check backend health from HAProxy container
|
|
incus exec oc-node-01:ffsdn-haproxy-52-01 -- \
|
|
curl -s http://localhost:8404/stats\;csv | grep backend_web
|
|
|
|
# Verify direct backend connectivity
|
|
incus exec oc-node-01:ffsdn-haproxy-52-01 -- curl -s http://10.10.10.60/
|
|
```
|
|
|
|
### Traffic not reaching backends
|
|
|
|
Even when the service is created and HAProxy is running, traffic may not
|
|
flow if:
|
|
|
|
- **ACL rules blocking traffic** — Aether creates per-instance ACLs with
|
|
default-deny. If the ACLs don't cover the traffic pattern, it gets
|
|
rejected. Check with `incus network acl show <remote>:<acl-name>`.
|
|
- **HAProxy config not reloaded** — click **Reload Config** in the Aether
|
|
UI or wait for automatic propagation.
|
|
- **Wrong backend port** — verify the backends are actually listening on
|
|
the configured port (80 in our case).
|
|
|
|
Test connectivity directly from a HAProxy container:
|
|
|
|
```bash
|
|
incus exec oc-node-01:ffsdn-haproxy-52-01 -- curl -s http://10.10.10.60/
|
|
```
|
|
|
|
### Sticky sessions not working
|
|
|
|
Sticky sessions only work with **HTTP** and **HTTPS Termination** modes.
|
|
They do NOT work with TCP or HTTPS Passthrough (HAProxy can't insert
|
|
cookies without inspecting HTTP content). Verify your service mode.
|
|
|
|
### Advanced options not taking effect
|
|
|
|
Advanced options (compression, HSTS, rate limiting, access control) only
|
|
apply to **HTTP** and **HTTPS Termination** modes. They have no effect on
|
|
TCP or HTTPS Passthrough services, as those operate at L4 without
|
|
inspecting HTTP content.
|
|
|
|
### Rate limiting blocking legitimate users
|
|
|
|
If users behind corporate NAT or VPN share a single IP, they collectively
|
|
count against the per-IP rate limit. A web page loading 20 images generates
|
|
21 requests. Start with higher limits (500+) and reduce based on monitoring.
|
|
|
|
### Session authentication issues
|
|
|
|
Aether's HAProxy endpoints use session authentication with CSRF protection.
|
|
**curl-based session auth is unreliable** — use Playwright browser automation
|
|
for reliable interaction (see `incusos/helpers/aether-browser`).
|
|
|
|
Common issues with curl-based auth:
|
|
1. **CSRF 403**: The CSRF token binding requires proper browser cookie handling.
|
|
Playwright handles this automatically.
|
|
2. **Login page at `/`**: The login form is at root (`/`), NOT `/login`.
|
|
`GET /login` returns 404. The form POSTs to `/login`.
|
|
3. **Session expiry**: Re-authenticate by re-running the Playwright login.
|
|
|
|
### Cleanup
|
|
|
|
To completely remove HAProxy infrastructure:
|
|
|
|
```bash
|
|
# Via Aether UI: HAProxy > select cluster > Delete Infrastructure
|
|
|
|
# Via session API (requires CSRF token from Playwright):
|
|
curl -sSk -b "$COOKIE_JAR" \
|
|
-H "X-CSRF-Token: ${CSRF}" \
|
|
-H "Referer: ${AETHER_URL}/haproxy" \
|
|
-X DELETE "${AETHER_URL}/haproxy/infrastructure/52"
|
|
|
|
# Remove the OVN load balancer (created manually, not cleaned by Aether)
|
|
incus network load-balancer delete oc-node-01:net-prod 192.168.103.200
|
|
|
|
# Clean up test backends
|
|
for name in nginx-lb-01 nginx-lb-02 nginx-lb-03; do
|
|
incus delete "oc-node-01:${name}" --force
|
|
done
|
|
```
|
|
|
|
**Warning**: Deleting infrastructure stops all traffic immediately.
|
|
It removes both HAProxy containers, all VIPs, all services, and
|
|
associated firewall rules.
|