incus-contrib/notes/api-interception-guide.md

# API Interception Guide: Observing Aether ↔ Incus Communication

How to capture, analyze, and understand the REST API calls between
Aether (SDN controller) and the Incus cluster at the wire level.

## Architecture

```
Aether (192.168.102.160)
  │  Client cert: CN=root@oc-server
  │  Subscribes to: /1.0/events (lifecycle stream)
  │
  ├──→ Incus API (https://192.168.102.140-142:8443)
  │    All instance/network/storage operations
  │
  └──→ Aether's own REST API (/api/*)
       Auth, clusters, ACLs only
       JWT-authenticated, separate from Incus

CLI (dev-vm-beelink)
  │  Client cert: CN=maarten@dev-vm-beelink
  │
  └──→ Incus API (same endpoints)

Cluster nodes (oc-node-01/02/03)
  │  Client certs: CN=root@oc-node-*
  │
  └──→ Inter-node API (internal cluster operations)
```

## Capture Methods

### Primary: Incus Event Stream

The `incus monitor` command subscribes to the Incus event websocket and
captures every API call, operation lifecycle change, and state transition.

```bash
# Quick capture (60 seconds)
incusos/helpers/incus-mitm --capture 60

# Long capture with custom output
incusos/helpers/incus-mitm --capture 300 --output /tmp/my-capture.json

# Live streaming (Ctrl+C to stop)
incusos/helpers/incus-mitm --live

# Filter for Aether only
incusos/helpers/incus-mitm --live --filter aether
```

**What gets captured:**
- Every REST API call (method, path, caller certificate)
- Operation lifecycle (Pending → Running → Success/Failure)
- Instance lifecycle events (created, started, stopped, deleted, etc.)
- Cluster-internal operations (node-to-node forwarding)

**What doesn't get captured:**
- Request/response bodies (only the path and method)
- Aether-internal operations (JWT API calls don't go through Incus)
- Timing/latency data (only timestamps)

### Secondary: Aether Go Log

Aether has a live log viewable at `/logs/live` in the web UI (sidebar →
"View Live GO Log"). This shows Aether's perspective: what it decides to
do and what errors it encounters.

Access via Playwright (session-authenticated route):
```bash
source env
NODE_PATH=~/node_modules node incusos/helpers/aether-browser screenshot /logs/live
```

### Tertiary: mitmproxy (TLS Interception)

For capturing full request/response bodies, a mitmproxy reverse proxy
can be deployed. This is complex due to mutual TLS and is only needed
if the event stream doesn't provide enough detail.

**Not implemented** — the event stream proved sufficient for understanding
the API protocol.

## Analyzing Captures

```bash
# Full analysis
incusos/helpers/incus-mitm --analyze /tmp/incus-events.json

# Show only Aether's calls
incusos/helpers/incus-mitm --analyze /tmp/incus-events.json --filter aether

# Identify callers by certificate
incusos/helpers/incus-mitm --callers /tmp/incus-events.json
```

## Certificate Fingerprints

Each caller is identified by their TLS client certificate fingerprint.
The event stream logs certificate matching, allowing us to identify who
makes each API call.

| Fingerprint Prefix | Subject | Role |
|---|---|---|
| `bb2dc9eac3d3...` | CN=maarten@dev-vm-beelink | CLI client |
| `6cfd2a1949a7...` | CN=root@oc-server | Aether SDN controller |
| `a68291ca3683...` | CN=root@oc-node-02 | Cluster inter-node |

Aether's cert (`root@oc-server`) is the key one — it identifies all
operations that Aether initiates against the Incus cluster.

## Captured API Patterns

### Instance Stop (CLI-initiated)

```
CLI  → PUT  /1.0/instances/<name>/state          (body: {"action":"stop"})
CLI  → GET  /1.0/operations/<uuid>                (poll for completion)
Incus → operation: Stopping instance [Pending → Running → Success]
Incus → lifecycle: instance-shutdown
```

### Instance Start (CLI-initiated)

```
CLI  → PUT  /1.0/instances/<name>/state          (body: {"action":"start"})
CLI  → GET  /1.0/operations/<uuid>
Incus → operation: Starting instance [Pending → Running → Success]
Incus → lifecycle: instance-started
```

### Snapshot Create

```
CLI  → POST /1.0/instances/<name>/snapshots       (body: {"name":"snap-name"})
CLI  → GET  /1.0/operations/<uuid>
Incus → operation: Snapshotting instance [Pending → Running → Success]
Incus → lifecycle: instance-snapshot-created
Aether → GET /1.0                                  (reacts to lifecycle event)
Aether → GET /1.0/instances/<snap-name>?project=default&recursion=1
```

### Snapshot Delete

```
CLI  → DELETE /1.0/instances/<name>/snapshots/<snap-name>
CLI  → GET   /1.0/operations/<uuid>
Incus → operation: Deleting snapshot [Pending → Running → Success]
Incus → lifecycle: instance-snapshot-deleted
Aether → GET /1.0                                  (reacts to lifecycle event)
Aether → GET /1.0/instances/<snap-name>?project=default&recursion=1
```

### Exec Command

```
CLI  → POST /1.0/instances/<name>/exec            (body: {"command":["hostname"],...})
Node → POST https://<target-node>:8443/1.0/instances/<name>/exec  (forwarded)
CLI  → GET  /1.0/operations/<uuid>/websocket?secret=<stdin>
CLI  → GET  /1.0/operations/<uuid>/websocket?secret=<stdout>
CLI  → GET  /1.0/operations/<uuid>/websocket?secret=<stderr>
CLI  → GET  /1.0/operations/<uuid>/websocket?secret=<control>
CLI  → GET  /1.0/operations/<uuid>                (poll completion)
Incus → operation: Executing command [Pending → Running → Success]
Incus → lifecycle: instance-exec
```

Note: exec uses 4 websocket connections (stdin, stdout, stderr, control).
The operation is forwarded to the target node where the instance runs.

### Network List

```
CLI  → GET /1.0/networks?filter=&recursion=1
```

### Instance Info

```
CLI  → GET /1.0/instances/<name>?recursion=1
```

## Key Findings

### 1. Aether Subscribes to Lifecycle Events

Aether maintains a persistent connection to the Incus event stream.
When any lifecycle event occurs (instance created, snapshot deleted, etc.),
Aether immediately queries the affected resource. This is how Aether
keeps its UI in sync with the cluster state.

Pattern: `lifecycle event → Aether GET /1.0 → Aether GET /1.0/instances/<name>`

### 2. Aether Uses Client Certificate Auth

Aether authenticates to Incus using a TLS client certificate
(`CN=root@oc-server`), not API tokens. This certificate is trusted by
all cluster members. The certificate is stored in the Aether VM
(likely at `/root/.config/incus/client.crt`).

### 3. Operations Are Forwarded

When an operation targets an instance on a different node, the receiving
node forwards the request internally. This is visible in the event stream
as a `POST https://<target-node>:8443/...` call from an "internal" caller.

### 4. Every CLI Command Starts with GET /1.0

The Incus CLI always fetches `/1.0` first (server info/capabilities)
before making any other call. It also opens an `/1.0/events` websocket
for real-time operation status updates.

### 5. Aether REST API Is Separate

Aether's own API (`/api/auth/token`, `/api/clusters`, `/api/clusters/*/rules`)
is completely separate from the Incus API. These calls go to Aether's
HTTPS server (port 8443) and use JWT authentication. They are NOT visible
in the Incus event stream.

Current Aether API coverage:
- `POST /api/auth/token` — JWT authentication
- `GET /api/clusters` — list managed clusters
- `CRUD /api/clusters/{id}/rules` — per-cluster ACL rules
- `CRUD /api/global/rules` — global ACL rules

All other Aether features (deploys, HAProxy, blueprints, instance
management) are UI-only and use session-authenticated routes with CSRF
protection. These features call the Incus API directly via Aether's
server-side code, visible in the event stream as calls from the
`root@oc-server` certificate.

## Reproducing the Capture

```bash
# 1. Start capture in one terminal
incusos/helpers/incus-mitm --live

# 2. In another terminal, trigger operations:
incus stop oc-node-01:<instance>
incus start oc-node-01:<instance>
incus snapshot create oc-node-01:<instance> test-snap
incus snapshot delete oc-node-01:<instance> test-snap

# 3. Or trigger via Aether UI:
#    - Deploy a container
#    - Create a blueprint
#    - Manage HAProxy
#    (All will appear in the event stream as Aether API calls)
```