256 lines
8.4 KiB
Markdown
256 lines
8.4 KiB
Markdown
# API Interception Guide: Observing Aether ↔ Incus Communication
|
|
|
|
How to capture, analyze, and understand the REST API calls between
|
|
Aether (SDN controller) and the Incus cluster at the wire level.
|
|
|
|
## Architecture
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
subgraph callers["API Callers"]
|
|
aether["Aether<br/>192.168.102.160<br/>CN=root@oc-server"]
|
|
cli["CLI<br/>dev-vm-beelink<br/>CN=maarten@dev-vm-beelink"]
|
|
nodes["Cluster Nodes<br/>oc-node-01/02/03<br/>CN=root@oc-node-*"]
|
|
end
|
|
|
|
incus["Incus API<br/>:8443 on .140-.142"]
|
|
aether_api["Aether REST API<br/>/api/* (JWT auth)"]
|
|
|
|
aether -->|"client cert<br/>+ /1.0/events"| incus
|
|
aether --> aether_api
|
|
cli -->|"client cert"| incus
|
|
nodes -->|"inter-node"| incus
|
|
|
|
classDef mgmt fill:#CC79A7,color:#fff,stroke:#a36088
|
|
classDef instance fill:#56B4E9,color:#fff,stroke:#3a8fbf
|
|
classDef network fill:#0072B2,color:#fff,stroke:#005a8e
|
|
classDef node fill:#009E73,color:#fff,stroke:#007a5e
|
|
|
|
class aether mgmt
|
|
class cli instance
|
|
class nodes node
|
|
class incus,aether_api network
|
|
|
|
style callers fill:#f5f5f5,stroke:#999
|
|
```
|
|
|
|
## Capture Methods
|
|
|
|
### Primary: Incus Event Stream
|
|
|
|
The `incus monitor` command subscribes to the Incus event websocket and
|
|
captures every API call, operation lifecycle change, and state transition.
|
|
|
|
```bash
|
|
# Quick capture (60 seconds)
|
|
incusos/helpers/incus-mitm --capture 60
|
|
|
|
# Long capture with custom output
|
|
incusos/helpers/incus-mitm --capture 300 --output /tmp/my-capture.json
|
|
|
|
# Live streaming (Ctrl+C to stop)
|
|
incusos/helpers/incus-mitm --live
|
|
|
|
# Filter for Aether only
|
|
incusos/helpers/incus-mitm --live --filter aether
|
|
```
|
|
|
|
**What gets captured:**
|
|
- Every REST API call (method, path, caller certificate)
|
|
- Operation lifecycle (Pending → Running → Success/Failure)
|
|
- Instance lifecycle events (created, started, stopped, deleted, etc.)
|
|
- Cluster-internal operations (node-to-node forwarding)
|
|
|
|
**What doesn't get captured:**
|
|
- Request/response bodies (only the path and method)
|
|
- Aether-internal operations (JWT API calls don't go through Incus)
|
|
- Timing/latency data (only timestamps)
|
|
|
|
### Secondary: Aether Go Log
|
|
|
|
Aether has a live log viewable at `/logs/live` in the web UI (sidebar →
|
|
"View Live GO Log"). This shows Aether's perspective: what it decides to
|
|
do and what errors it encounters.
|
|
|
|
Access via Playwright (session-authenticated route):
|
|
```bash
|
|
source env
|
|
NODE_PATH=~/node_modules node incusos/helpers/aether-browser screenshot /logs/live
|
|
```
|
|
|
|
### Tertiary: mitmproxy (TLS Interception)
|
|
|
|
For capturing full request/response bodies, a mitmproxy reverse proxy
|
|
can be deployed. This is complex due to mutual TLS and is only needed
|
|
if the event stream doesn't provide enough detail.
|
|
|
|
**Not implemented** — the event stream proved sufficient for understanding
|
|
the API protocol.
|
|
|
|
## Analyzing Captures
|
|
|
|
```bash
|
|
# Full analysis
|
|
incusos/helpers/incus-mitm --analyze /tmp/incus-events.json
|
|
|
|
# Show only Aether's calls
|
|
incusos/helpers/incus-mitm --analyze /tmp/incus-events.json --filter aether
|
|
|
|
# Identify callers by certificate
|
|
incusos/helpers/incus-mitm --callers /tmp/incus-events.json
|
|
```
|
|
|
|
## Certificate Fingerprints
|
|
|
|
Each caller is identified by their TLS client certificate fingerprint.
|
|
The event stream logs certificate matching, allowing us to identify who
|
|
makes each API call.
|
|
|
|
| Fingerprint Prefix | Subject | Role |
|
|
|---|---|---|
|
|
| `bb2dc9eac3d3...` | CN=maarten@dev-vm-beelink | CLI client |
|
|
| `6cfd2a1949a7...` | CN=root@oc-server | Aether SDN controller |
|
|
| `a68291ca3683...` | CN=root@oc-node-02 | Cluster inter-node |
|
|
|
|
Aether's cert (`root@oc-server`) is the key one — it identifies all
|
|
operations that Aether initiates against the Incus cluster.
|
|
|
|
## Captured API Patterns
|
|
|
|
### Instance Stop (CLI-initiated)
|
|
|
|
```
|
|
CLI → PUT /1.0/instances/<name>/state (body: {"action":"stop"})
|
|
CLI → GET /1.0/operations/<uuid> (poll for completion)
|
|
Incus → operation: Stopping instance [Pending → Running → Success]
|
|
Incus → lifecycle: instance-shutdown
|
|
```
|
|
|
|
### Instance Start (CLI-initiated)
|
|
|
|
```
|
|
CLI → PUT /1.0/instances/<name>/state (body: {"action":"start"})
|
|
CLI → GET /1.0/operations/<uuid>
|
|
Incus → operation: Starting instance [Pending → Running → Success]
|
|
Incus → lifecycle: instance-started
|
|
```
|
|
|
|
### Snapshot Create
|
|
|
|
```
|
|
CLI → POST /1.0/instances/<name>/snapshots (body: {"name":"snap-name"})
|
|
CLI → GET /1.0/operations/<uuid>
|
|
Incus → operation: Snapshotting instance [Pending → Running → Success]
|
|
Incus → lifecycle: instance-snapshot-created
|
|
Aether → GET /1.0 (reacts to lifecycle event)
|
|
Aether → GET /1.0/instances/<snap-name>?project=default&recursion=1
|
|
```
|
|
|
|
### Snapshot Delete
|
|
|
|
```
|
|
CLI → DELETE /1.0/instances/<name>/snapshots/<snap-name>
|
|
CLI → GET /1.0/operations/<uuid>
|
|
Incus → operation: Deleting snapshot [Pending → Running → Success]
|
|
Incus → lifecycle: instance-snapshot-deleted
|
|
Aether → GET /1.0 (reacts to lifecycle event)
|
|
Aether → GET /1.0/instances/<snap-name>?project=default&recursion=1
|
|
```
|
|
|
|
### Exec Command
|
|
|
|
```
|
|
CLI → POST /1.0/instances/<name>/exec (body: {"command":["hostname"],...})
|
|
Node → POST https://<target-node>:8443/1.0/instances/<name>/exec (forwarded)
|
|
CLI → GET /1.0/operations/<uuid>/websocket?secret=<stdin>
|
|
CLI → GET /1.0/operations/<uuid>/websocket?secret=<stdout>
|
|
CLI → GET /1.0/operations/<uuid>/websocket?secret=<stderr>
|
|
CLI → GET /1.0/operations/<uuid>/websocket?secret=<control>
|
|
CLI → GET /1.0/operations/<uuid> (poll completion)
|
|
Incus → operation: Executing command [Pending → Running → Success]
|
|
Incus → lifecycle: instance-exec
|
|
```
|
|
|
|
Note: exec uses 4 websocket connections (stdin, stdout, stderr, control).
|
|
The operation is forwarded to the target node where the instance runs.
|
|
|
|
### Network List
|
|
|
|
```
|
|
CLI → GET /1.0/networks?filter=&recursion=1
|
|
```
|
|
|
|
### Instance Info
|
|
|
|
```
|
|
CLI → GET /1.0/instances/<name>?recursion=1
|
|
```
|
|
|
|
## Key Findings
|
|
|
|
### 1. Aether Subscribes to Lifecycle Events
|
|
|
|
Aether maintains a persistent connection to the Incus event stream.
|
|
When any lifecycle event occurs (instance created, snapshot deleted, etc.),
|
|
Aether immediately queries the affected resource. This is how Aether
|
|
keeps its UI in sync with the cluster state.
|
|
|
|
Pattern: `lifecycle event → Aether GET /1.0 → Aether GET /1.0/instances/<name>`
|
|
|
|
### 2. Aether Uses Client Certificate Auth
|
|
|
|
Aether authenticates to Incus using a TLS client certificate
|
|
(`CN=root@oc-server`), not API tokens. This certificate is trusted by
|
|
all cluster members. The certificate is stored in the Aether VM
|
|
(likely at `/root/.config/incus/client.crt`).
|
|
|
|
### 3. Operations Are Forwarded
|
|
|
|
When an operation targets an instance on a different node, the receiving
|
|
node forwards the request internally. This is visible in the event stream
|
|
as a `POST https://<target-node>:8443/...` call from an "internal" caller.
|
|
|
|
### 4. Every CLI Command Starts with GET /1.0
|
|
|
|
The Incus CLI always fetches `/1.0` first (server info/capabilities)
|
|
before making any other call. It also opens an `/1.0/events` websocket
|
|
for real-time operation status updates.
|
|
|
|
### 5. Aether REST API Is Separate
|
|
|
|
Aether's own API (`/api/auth/token`, `/api/clusters`, `/api/clusters/*/rules`)
|
|
is completely separate from the Incus API. These calls go to Aether's
|
|
HTTPS server (port 8443) and use JWT authentication. They are NOT visible
|
|
in the Incus event stream.
|
|
|
|
Current Aether API coverage:
|
|
- `POST /api/auth/token` — JWT authentication
|
|
- `GET /api/clusters` — list managed clusters
|
|
- `CRUD /api/clusters/{id}/rules` — per-cluster ACL rules
|
|
- `CRUD /api/global/rules` — global ACL rules
|
|
|
|
All other Aether features (deploys, HAProxy, blueprints, instance
|
|
management) are UI-only and use session-authenticated routes with CSRF
|
|
protection. These features call the Incus API directly via Aether's
|
|
server-side code, visible in the event stream as calls from the
|
|
`root@oc-server` certificate.
|
|
|
|
## Reproducing the Capture
|
|
|
|
```bash
|
|
# 1. Start capture in one terminal
|
|
incusos/helpers/incus-mitm --live
|
|
|
|
# 2. In another terminal, trigger operations:
|
|
incus stop oc-node-01:<instance>
|
|
incus start oc-node-01:<instance>
|
|
incus snapshot create oc-node-01:<instance> test-snap
|
|
incus snapshot delete oc-node-01:<instance> test-snap
|
|
|
|
# 3. Or trigger via Aether UI:
|
|
# - Deploy a container
|
|
# - Create a blueprint
|
|
# - Manage HAProxy
|
|
# (All will appear in the event stream as Aether API calls)
|
|
```
|