# HAProxy Load Balancing — Aether-Managed HA Proxy on OVN

Aether includes a full HAProxy management system that deploys a highly
available load balancer pair on OVN networks. It handles image builds,
HA pair deployment, VIP management, L7 service configuration, and health
monitoring — all through its web UI and session-authenticated API.

This guide covers deploying HAProxy infrastructure on the lab cluster,
creating test backends, configuring services, and verifying load balancing.

## Architecture

```mermaid
flowchart TD
    client(["Client / LAN"])
    vip["VIP 192.168.103.200"]
    ovnlb("OVN Load Balancer")
    ha1["HAProxy 01<br/>10.10.10.50"]
    ha2["HAProxy 02<br/>10.10.10.51"]
    ng1["nginx-01 · .60"]
    ng2["nginx-02 · .61"]
    ng3["nginx-03 · .62"]

    client --> vip
    vip --> ovnlb
    ovnlb --> ha1 & ha2
    ha1 & ha2 --> ng1 & ng2 & ng3

    classDef external fill:#f5f5f5,color:#333,stroke:#999
    classDef network fill:#0072B2,color:#fff,stroke:#005a8e
    classDef lb fill:#E69F00,color:#fff,stroke:#b87d00
    classDef instance fill:#56B4E9,color:#fff,stroke:#3a8fbf

    class client external
    class vip,ovnlb network
    class ha1,ha2 lb
    class ng1,ng2,ng3 instance
```

### OVN native LB vs HAProxy

| Feature | OVN native LB | Aether HAProxy |
|---------|---------------|----------------|
| Layer | L4 (TCP/UDP) | L4 + L7 (HTTP/HTTPS) |
| Health checks | None | HTTP, TCP, configurable interval |
| SSL termination | No | Yes (HTTPS Termination mode) |
| Sticky sessions | No | Yes (cookie-based) |
| Load balancing | Connection hash | Round robin, least conn, source hash, first available |
| SNI routing | No | Yes (multiple domains on same VIP:port) |
| Compression | No | Yes (gzip for text content) |
| Rate limiting | No | Yes (per-IP) |
| Management | `incus network load-balancer` CLI | Aether UI + session API |
| HA | Single OVN LB (distributed) | HA pair with OVN LB failover |

Use OVN native LB for simple L4 distribution. Use HAProxy when you need
health checks, L7 features, SSL, sticky sessions, or advanced traffic management.

## Prerequisites

- Incus cluster registered in Aether with OVN networking configured
- OVN network with a physical UPLINK (for VIP addresses)
- Available IP addresses on the OVN network for HAProxy instances
- Available VIP address(es) from the UPLINK range
- Internet access from the cluster (for HAProxy image builds)
- `incus` CLI configured with cluster remote
- Aether admin access (Full Admin or LB Admin role)

### Resource requirements

| Resource | Per HAProxy instance | Notes |
|----------|---------------------|-------|
| CPU | 2 cores (default) | HAProxy is very CPU-efficient |
| Memory | 1 GiB (default) | Increase for many concurrent connections |
| Disk | Minimal | Container image, no persistent storage needed |

## Lab details

| Setting | Value |
|---------|-------|
| Cluster | oc-lab-cluster (ID: 52) |
| OVN network | net-prod (10.10.10.0/24) |
| UPLINK | UPLINK (physical) |
| VIP range | 192.168.103.200-210 |
| HAProxy 01 IP | 10.10.10.50 |
| HAProxy 02 IP | 10.10.10.51 |
| Backend IPs | 10.10.10.60, 10.10.10.61, 10.10.10.62 |
| HAProxy containers | ffsdn-haproxy-52-01, ffsdn-haproxy-52-02 |
| Aether URL | https://192.168.102.160:8443 |
| Cluster remote | oc-node-01 |

## Automated path

The `incusos/deploy-haproxy` script automates the entire deployment:

```bash
# Check prerequisites
deploy-haproxy --doctor

# Preview deployment
deploy-haproxy --deploy --dry-run

# Full deployment (backends + image + infrastructure + service)
deploy-haproxy --deploy

# Check health
deploy-haproxy --status

# Clean removal
deploy-haproxy --cleanup
```

The manual steps below are for reference, troubleshooting, and understanding
what the script does.

## Step 1: Deploy test backends

Create 3 nginx containers on net-prod to serve as load-balanced backends.
Each gets a unique index page so we can verify traffic distribution.

**Important**: The backend IPs (10.10.10.60-62) must not already be in use
on the OVN network. Check the Aether UI (HAProxy > select cluster) for
used IPs before deploying, or use `deploy-haproxy --doctor` which checks
for you.

```bash
REMOTE="oc-node-01"

for i in 60 61 62; do
    NAME="nginx-lb-$(printf '%02d' $((i - 59)))"
    IP="10.10.10.${i}"

    # Launch container on OVN network with static IP via device key
    incus launch images:debian/12 "${REMOTE}:${NAME}" \
        --network net-prod \
        --storage local \
        -d root,size=2GiB \
        -d eth0,ipv4.address="${IP}" \
        -c limits.cpu=1 \
        -c limits.memory=256MiB

    # Wait for container agent
    echo "Waiting for ${NAME}..."
    while ! incus exec "${REMOTE}:${NAME}" -- true 2>/dev/null; do
        sleep 2
    done

    # Install nginx + curl, create unique index page
    incus exec "${REMOTE}:${NAME}" -- bash -c "
        export DEBIAN_FRONTEND=noninteractive
        apt-get update -qq
        apt-get install -y -qq nginx curl 2>&1 | tail -1

        cat > /var/www/html/index.html << HTML
<html><body>
<h1>Backend: ${NAME}</h1>
<p>IP: ${IP}</p>
</body></html>
HTML

        systemctl enable nginx
        systemctl start nginx
    "

    echo "Backend ${NAME} deployed at ${IP}"
done
```

Verify all backends respond (curling from one backend to the others
confirms OVN cross-container connectivity):

```bash
# Wait for static IPs to take effect
sleep 5

for i in 60 61 62; do
    incus exec "${REMOTE}:nginx-lb-01" -- \
        curl -s --connect-timeout 3 "http://10.10.10.${i}/" \
        | grep -o 'Backend: [^<]*'
done
# Expected:
# Backend: nginx-lb-01
# Backend: nginx-lb-02
# Backend: nginx-lb-03
```

If curl fails, the static IP may not have taken effect yet. Verify with
`incus exec "${REMOTE}:nginx-lb-01" -- ip addr show eth0`.

## Step 2: Build HAProxy image

Aether builds a base HAProxy container image and stores it internally.
The image must be pushed to each target cluster before infrastructure
can be deployed.

**Note**: The HAProxy management features are not in Aether's JWT API
(`/api/swagger.yaml`). They use session-authenticated routes discovered
from the UI JavaScript. The script uses session auth automatically.

### Via Aether UI

1. Navigate to **HAProxy** in the main menu
2. In **HAProxy Base Images**, click **Build New Image**
3. Select **Build on Cluster**: `oc-lab-cluster`
4. Enter **Version**: `1.0.0`
5. Click **Build Image**
6. Wait 5-10 minutes for the build to complete
7. Click **Push to Cluster** and select `oc-lab-cluster`
8. Click **Set Current** on the new image

### Via session API

The build process uses session cookies + CSRF token authentication:

```bash
AETHER_URL="https://192.168.102.160:8443"
COOKIE_JAR="/tmp/aether-cookies.txt"

# Step 1: Get login page and extract CSRF token from hidden form field
# Note: the login form is at "/" (root), NOT "/login". GET /login returns 404.
CSRF=$(curl -sSk -c "$COOKIE_JAR" "${AETHER_URL}/" \
    | grep -o 'name="csrf_token" value="[^"]*"' \
    | sed 's/.*value="//;s/"//')

# Step 2: Login with CSRF token (form-urlencoded, NOT JSON)
curl -sSk -b "$COOKIE_JAR" -c "$COOKIE_JAR" \
    -X POST "${AETHER_URL}/login" \
    -d "csrf_token=${CSRF}&username=admin&password=${AETHER_ADMIN_PASSWORD}" \
    -L -o /dev/null

# Step 3: Get fresh CSRF token for API calls
CSRF=$(curl -sSk -b "$COOKIE_JAR" "${AETHER_URL}/haproxy" \
    | grep -o 'name="csrf_token" value="[^"]*"' \
    | sed 's/.*value="//;s/"//')

# Step 4: Build image
curl -sSk -b "$COOKIE_JAR" \
    -H "X-CSRF-Token: ${CSRF}" \
    -H "Referer: ${AETHER_URL}/haproxy" \
    -X POST "${AETHER_URL}/haproxy/image/build" \
    -H "Content-Type: application/json" \
    -d '{"cluster_id": 52, "version": "1.0.0"}'

# Step 5: Poll build status until complete
while true; do
    STATUS=$(curl -sSk -b "$COOKIE_JAR" \
        -H "X-CSRF-Token: ${CSRF}" \
        "${AETHER_URL}/haproxy/image/build-status" \
        | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('status','unknown'))")
    echo "Build status: $STATUS"
    [[ "$STATUS" == "completed" || "$STATUS" == "failed" ]] && break
    sleep 10
done

# Step 6: Get image ID and push to cluster
IMAGE_ID=$(curl -sSk -b "$COOKIE_JAR" \
    -H "X-CSRF-Token: ${CSRF}" \
    "${AETHER_URL}/haproxy/images" \
    | python3 -c "import sys,json; imgs=json.load(sys.stdin); print(imgs[-1]['id'] if imgs else '')")

curl -sSk -b "$COOKIE_JAR" \
    -H "X-CSRF-Token: ${CSRF}" \
    -H "Referer: ${AETHER_URL}/haproxy" \
    -X POST "${AETHER_URL}/haproxy/image/push" \
    -H "Content-Type: application/json" \
    -d "{\"image_id\": ${IMAGE_ID}, \"cluster_id\": 52}"

# Step 7: Set as current version
curl -sSk -b "$COOKIE_JAR" \
    -H "X-CSRF-Token: ${CSRF}" \
    -H "Referer: ${AETHER_URL}/haproxy" \
    -X POST "${AETHER_URL}/haproxy/image/set-current" \
    -H "Content-Type: application/json" \
    -d "{\"image_id\": ${IMAGE_ID}}"
```

**Critical**: The CSRF token comes from a hidden HTML form field
(`<input name="csrf_token" value="...">`), NOT from the cookie value.
The cookie contains the session ID; the form field contains the CSRF token.

## Step 3: Deploy HAProxy infrastructure

Deploy the HA pair on the cluster, then create the OVN load balancer.

### Via Aether UI

1. Navigate to **HAProxy** > select **oc-lab-cluster** from dropdown
2. Click **Deploy HAProxy Infrastructure**
3. Fill in:
   - **OVN Network**: `net-prod`
   - **Load Balancer VIP**: `192.168.103.200` (from UPLINK range dropdown)
   - **HAProxy 01 IP**: `10.10.10.50`
   - **HAProxy 02 IP**: `10.10.10.51`
   - **CPU Limit**: `2`
   - **Memory Limit**: `1GB`
4. Click **Deploy**

### Via session API

```bash
curl -sSk -b "$COOKIE_JAR" \
    -H "X-CSRF-Token: ${CSRF}" \
    -H "Referer: ${AETHER_URL}/haproxy" \
    -X POST "${AETHER_URL}/haproxy/infrastructure/deploy" \
    -H "Content-Type: application/json" \
    -d '{
        "cluster_id": 52,
        "ovn_network": "net-prod",
        "lb_vip": "192.168.103.200",
        "haproxy_01_ip": "10.10.10.50",
        "haproxy_02_ip": "10.10.10.51",
        "cpu_limit": "2",
        "memory_limit": "1GB"
    }'
```

This creates:
- Two containers: `ffsdn-haproxy-52-01` and `ffsdn-haproxy-52-02`
- Per-instance ACL rules allowing traffic from the VIP
- A default VIP entry at 192.168.103.200

**Important**: Aether does NOT create the OVN load balancer automatically.
You must create it manually after infrastructure deployment:

```bash
REMOTE="oc-node-01"

# Create OVN load balancer for the VIP
incus network load-balancer create "${REMOTE}:net-prod" 192.168.103.200 \
    --description "FFSDN HAProxy Load Balancer"

# Add both HAProxy instances as backends
incus network load-balancer backend add "${REMOTE}:net-prod" 192.168.103.200 \
    haproxy-01 10.10.10.50 80
incus network load-balancer backend add "${REMOTE}:net-prod" 192.168.103.200 \
    haproxy-02 10.10.10.51 80

# Map port 80 to both backends
incus network load-balancer port add "${REMOTE}:net-prod" 192.168.103.200 \
    tcp 80 haproxy-01,haproxy-02

# Verify
incus network load-balancer list "${REMOTE}:net-prod"
```

Verify infrastructure:

```bash
# Check via API
curl -sSk -b "$COOKIE_JAR" \
    -H "X-CSRF-Token: ${CSRF}" \
    "${AETHER_URL}/haproxy/infrastructure/52"

# Check containers exist on cluster
incus list oc-node-01: --format csv -c n | grep ffsdn-haproxy
# ffsdn-haproxy-52-01
# ffsdn-haproxy-52-02
```

### Fix ACL rules for external access

Aether auto-generates per-instance ACLs that only allow ingress from
`source: 192.168.103.200` (the VIP address). This works for traffic
originating inside the OVN network — OVN hairpin-SNATs the source to
the router IP (.200) when load-balancing back into the same network.
However, external traffic (from outside OVN) preserves the original
client IP, which doesn't match the ACL and gets **rejected**.

Add a broader ingress rule to each HAProxy container's ACL:

```bash
REMOTE="oc-node-01"
CLUSTER_ID=52

for n in 01 02; do
    ACL_NAME="${CLUSTER_ID}-ffsdn-haproxy-${CLUSTER_ID}-${n}-aether-acl"
    incus network acl rule add "${REMOTE}:${ACL_NAME}" ingress \
        action=allow protocol=tcp destination_port=80,443 \
        description="Allow external clients to reach HAProxy"
done
```

**Why is this needed?** OVN load balancers perform DNAT (rewriting the
destination IP from the VIP to a backend), but for external-to-internal
traffic they do NOT SNAT the source. The packet arrives at the HAProxy
container with the external client's IP as source, which Aether's narrow
ACL rule rejects. For internal traffic, OVN *does* hairpin-SNAT the source
to .200 (to prevent asymmetric routing within the same logical switch),
which is why it works from inside OVN without this fix.

**Note**: This ACL modification survives container restarts and
reconfigurations. However, if you delete and redeploy the HAProxy
infrastructure, Aether recreates the ACLs from scratch, so you'll
need to re-apply this fix. Check future Aether versions — this may
be fixed upstream.

## Step 4: Create service

Configure HAProxy to load-balance traffic across the nginx backends.

**Important: TCP vs HTTP mode** — For nginx backends, use **TCP mode**.
HTTP mode generates `option httpchk` which sends `OPTIONS /` requests.
Nginx returns `405 Not Allowed` for OPTIONS, causing HAProxy to mark
all backends as DOWN. TCP mode uses simple TCP health checks (L4OK).

### Via Aether UI

1. In the **Services** section, click **+ Add Service**
2. Fill in:
   - **Service Name**: `web-http`
   - **Description**: `Load balancer for nginx test backends`
   - **VIP**: `192.168.103.200 - Default`
   - **Listen Port**: `80`
   - **Mode**: `TCP (Layer 4)`
   - **Balance Method**: `Round Robin`
   - **Health Check**: enabled
3. Add backend servers:
   - `nginx-lb-01` / `10.10.10.60` / port `80` / weight `100`
   - `nginx-lb-02` / `10.10.10.61` / port `80` / weight `100`
   - `nginx-lb-03` / `10.10.10.62` / port `80` / weight `100`
4. Click **Create Service**

### Via session API

**Note**: The `vip_id` is the numeric VIP ID from the infrastructure
deployment (visible in the API response or the Aether UI). For the
default VIP created during deployment, retrieve it from
`GET /haproxy/infrastructure/52` → `vips[0].id`.

```bash
curl -sSk -b "$COOKIE_JAR" \
    -H "X-CSRF-Token: ${CSRF}" \
    -H "Referer: ${AETHER_URL}/haproxy" \
    -X POST "${AETHER_URL}/haproxy/service" \
    -H "Content-Type: application/json" \
    -d '{
        "vip_id": 42,
        "name": "web-http",
        "description": "HTTP load balancer for nginx test backends",
        "hostname": "",
        "listen_port": 80,
        "mode": "tcp",
        "balance_method": "roundrobin",
        "health_check_enabled": true,
        "session_persistence": false,
        "backends": [
            {"name": "nginx-lb-01", "target_ip": "10.10.10.60", "target_port": 80, "weight": 100},
            {"name": "nginx-lb-02", "target_ip": "10.10.10.61", "target_port": 80, "weight": 100},
            {"name": "nginx-lb-03", "target_ip": "10.10.10.62", "target_port": 80, "weight": 100}
        ]
    }'
```

**Note**: After creating or editing a service, Aether may need to reload
the HAProxy configuration on both instances. If the service doesn't respond
immediately, click **Reload Config** in the Aether UI, or wait a few seconds
for automatic propagation.

## Step 5: Test load balancing

### Internal connectivity (via HAProxy localhost)

Test HAProxy directly by curling localhost on each container. This
bypasses the OVN load balancer and tests HAProxy's backend routing:

```bash
# Direct test on HAProxy 01 (bypasses OVN LB)
for i in $(seq 1 6); do
    incus exec oc-node-01:ffsdn-haproxy-52-01 -- \
        curl -s http://localhost/ | grep -o 'Backend: [^<]*'
done
# Expected: nginx-lb-01, nginx-lb-02, nginx-lb-03 in rotation

# Direct test on HAProxy 02
incus exec oc-node-01:ffsdn-haproxy-52-02 -- curl -s http://localhost/
# Expected: response from one of the backends
```

**Note**: Curling the VIP (192.168.103.200) from within the OVN network
does NOT work reliably. The manually-created OVN load balancer handles
external-to-internal traffic but does not hairpin for internal-to-VIP
requests from the same logical switch. Use localhost for internal tests.

### External connectivity (from outside OVN)

After applying the ACL fix in Step 3, the VIP is reachable from any
machine on the LAN:

```bash
# Single request from external machine
curl -s http://192.168.103.200/
# Expected: response from one of the backends

# Verify round-robin across all backends
for i in $(seq 1 6); do
    curl -s http://192.168.103.200/ | grep -o '<h1>[^<]*</h1>'
done
# Expected: all three backends appear in rotation
```

If external access fails with "Connection refused", the ACL fix was
not applied. See Step 3 "Fix ACL rules for external access".

### Backend failover

```bash
# Stop one backend
incus stop oc-node-01:nginx-lb-03

# Wait for health check to detect (default: 5 seconds)
sleep 10

# Verify traffic only goes to remaining backends (external)
for i in $(seq 1 6); do
    curl -s http://192.168.103.200/ | grep -o '<h1>[^<]*</h1>'
done
# Expected: only nginx-lb-01 and nginx-lb-02

# Restart the backend
incus start oc-node-01:nginx-lb-03

# Wait for health check to re-add
sleep 10

# Verify all three are back in rotation
for i in $(seq 1 6); do
    curl -s http://192.168.103.200/ | grep -o '<h1>[^<]*</h1>'
done
```

### HAProxy failover

```bash
# Stop one HAProxy container
incus stop oc-node-01:ffsdn-haproxy-52-02

# Wait for OVN to detect the dead backend
sleep 15

# Some requests will succeed (routed to HAProxy 01), others may time out
curl -s --connect-timeout 10 http://192.168.103.200/
# Expected: response from one of the backends (via HAProxy 01)

# Restart
incus start oc-node-01:ffsdn-haproxy-52-02
```

**Note**: OVN load balancers do NOT have active health checking. When one
HAProxy instance is stopped, the OVN LB continues to send ~50% of
connections to the dead backend, causing those connections to time out.
After 10-15 seconds OVN may detect the dead backend, but failover is
not instant. For production use, consider Keepalived mode which provides
true active/standby VIP failover with VRRP.

## Monitoring

### Health and statistics

Each service has a Health & Stats view accessible via the Aether UI
(**Services** > **Health & Stats** button) or the session API:

```bash
curl -sSk -b "$COOKIE_JAR" \
    -H "X-CSRF-Token: ${CSRF}" \
    "${AETHER_URL}/haproxy/service/health-stats?cluster_id=52&service_name=web-http"
```

### Statistics fields

**Backend server statistics** (from HAProxy health checks):

| Field | Description |
|-------|-------------|
| Status | `UP` (healthy) or `DOWN` (removed from rotation) |
| Sessions | Current active / maximum seen |
| Bytes In/Out | Total traffic to/from this backend |
| Weight | Configured load distribution weight |
| Last Check | Most recent health check result + response time (ms) |

**Frontend statistics** (combined for all services on the same port):

| Field | Description |
|-------|-------------|
| Current Sessions | Active client connections |
| Max Sessions | Highest concurrent sessions seen |
| Total Sessions | Cumulative since HAProxy started |
| Session Rate | New sessions per second |
| Bytes In/Out | Total client traffic |

**HAProxy node resources** (per container):

| Field | Description | Color thresholds |
|-------|-------------|------------------|
| CPU Usage | Current utilization % | Green <50%, Orange 50-79%, Red 80%+ |
| Memory Usage | Current utilization | Green <50%, Orange 50-79%, Red 80%+ |

## Keepalived mode

For clusters without OVN networking, Aether supports Keepalived-based
HAProxy deployment. Instead of an OVN load balancer, Keepalived provides
VIP failover between the two HAProxy instances using VRRP.

Key differences:
- **No OVN required** — works with bridge or macvlan networking
- **VRRP failover** — active/standby instead of OVN's active/active LB
- **Direct VIP** — the VIP floats between HAProxy instances directly

This mode is selected during infrastructure deployment when choosing a
non-OVN network. The service configuration is identical regardless of
the underlying HA mechanism.

## Troubleshooting

### Image build fails

```bash
# Check build status
curl -sSk -b "$COOKIE_JAR" \
    -H "X-CSRF-Token: ${CSRF}" \
    "${AETHER_URL}/haproxy/image/build-status"
```

**Common causes:**
- No internet access from the cluster (package downloads fail)
- Cluster not reachable from Aether
- Insufficient disk space for image build

**Fix:** Ensure the cluster nodes can reach the internet. Try building
on a different cluster if available.

### Infrastructure deployment fails

**"No Image" in cluster dropdown:** The HAProxy image hasn't been pushed
to this cluster. Build an image (Step 2), then push it to the cluster
and set it as current.

**Containers not starting:** Check the Incus cluster has sufficient
resources (CPU, memory) for two additional containers.

```bash
# Check container state
incus list oc-node-01: -c ns --format csv | grep ffsdn-haproxy
```

### HAProxy container not starting after deploy

Aether may deploy both containers but only start one. This was observed
with `ffsdn-haproxy-52-01` remaining in STOPPED state after deployment.

```bash
# Check both containers
incus list oc-node-01: -f compact -c nsN4 | grep ffsdn-haproxy

# Start the stopped container
incus start oc-node-01:ffsdn-haproxy-52-01
```

### Service not responding

**VIP not reachable from outside OVN ("Connection refused"):**
Aether auto-generates per-instance ACL rules that only allow ingress
from `source: 192.168.103.200` (the VIP address). External traffic
preserves the original client IP as source, which the ACL rejects.
Apply the ACL fix from Step 3:

```bash
# Diagnose: check the ACL rules
incus network acl show oc-node-01:52-ffsdn-haproxy-52-01-aether-acl

# If the only ingress rule has source: 192.168.103.200, add a broader rule:
for n in 01 02; do
    incus network acl rule add "oc-node-01:52-ffsdn-haproxy-52-${n}-aether-acl" \
        ingress action=allow protocol=tcp destination_port=80,443 \
        description="Allow external clients to reach HAProxy"
done

# Verify it works now:
curl -s http://192.168.103.200/
```

**Why internal access works but external doesn't:** OVN hairpin-SNATs
traffic that originates inside the OVN network and gets load-balanced
back into the same network (source becomes .200, matching the ACL).
External traffic doesn't get SNAT'd, so the original client IP doesn't
match the narrow ACL rule.

**Backends showing DOWN in health stats (HTTP mode):**
When using **HTTP mode**, Aether generates `option httpchk` which sends
an `OPTIONS /` request. Nginx returns `405 Not Allowed`, marking backends
DOWN. **Use TCP mode** for nginx backends, or check the HAProxy stats:
```bash
# Check backend health from HAProxy container
incus exec oc-node-01:ffsdn-haproxy-52-01 -- \
    curl -s http://localhost:8404/stats\;csv | grep backend_web

# Verify direct backend connectivity
incus exec oc-node-01:ffsdn-haproxy-52-01 -- curl -s http://10.10.10.60/
```

### Traffic not reaching backends

Even when the service is created and HAProxy is running, traffic may not
flow if:

- **ACL rules blocking traffic** — Aether creates per-instance ACLs with
  default-deny. If the ACLs don't cover the traffic pattern, it gets
  rejected. Check with `incus network acl show <remote>:<acl-name>`.
- **HAProxy config not reloaded** — click **Reload Config** in the Aether
  UI or wait for automatic propagation.
- **Wrong backend port** — verify the backends are actually listening on
  the configured port (80 in our case).

Test connectivity directly from a HAProxy container:

```bash
incus exec oc-node-01:ffsdn-haproxy-52-01 -- curl -s http://10.10.10.60/
```

### Sticky sessions not working

Sticky sessions only work with **HTTP** and **HTTPS Termination** modes.
They do NOT work with TCP or HTTPS Passthrough (HAProxy can't insert
cookies without inspecting HTTP content). Verify your service mode.

### Advanced options not taking effect

Advanced options (compression, HSTS, rate limiting, access control) only
apply to **HTTP** and **HTTPS Termination** modes. They have no effect on
TCP or HTTPS Passthrough services, as those operate at L4 without
inspecting HTTP content.

### Rate limiting blocking legitimate users

If users behind corporate NAT or VPN share a single IP, they collectively
count against the per-IP rate limit. A web page loading 20 images generates
21 requests. Start with higher limits (500+) and reduce based on monitoring.

### Session authentication issues

Aether's HAProxy endpoints use session authentication with CSRF protection.
**curl-based session auth is unreliable** — use Playwright browser automation
for reliable interaction (see `incusos/helpers/aether-browser`).

Common issues with curl-based auth:
1. **CSRF 403**: The CSRF token binding requires proper browser cookie handling.
   Playwright handles this automatically.
2. **Login page at `/`**: The login form is at root (`/`), NOT `/login`.
   `GET /login` returns 404. The form POSTs to `/login`.
3. **Session expiry**: Re-authenticate by re-running the Playwright login.

### Cleanup

To completely remove HAProxy infrastructure:

```bash
# Via Aether UI: HAProxy > select cluster > Delete Infrastructure

# Via session API (requires CSRF token from Playwright):
curl -sSk -b "$COOKIE_JAR" \
    -H "X-CSRF-Token: ${CSRF}" \
    -H "Referer: ${AETHER_URL}/haproxy" \
    -X DELETE "${AETHER_URL}/haproxy/infrastructure/52"

# Remove the OVN load balancer (created manually, not cleaned by Aether)
incus network load-balancer delete oc-node-01:net-prod 192.168.103.200

# Clean up test backends
for name in nginx-lb-01 nginx-lb-02 nginx-lb-03; do
    incus delete "oc-node-01:${name}" --force
done
```

**Warning**: Deleting infrastructure stops all traffic immediately.
It removes both HAProxy containers, all VIPs, all services, and
associated firewall rules.