maarten
/
incus-contrib
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
incus-contrib/notes/api-interception-guide.md

8.1 KiB
Raw Blame History

API Interception Guide: Observing Aether ↔ Incus Communication

How to capture, analyze, and understand the REST API calls between Aether (SDN controller) and the Incus cluster at the wire level.

Architecture

Aether (192.168.102.160)
  │  Client cert: CN=root@oc-server
  │  Subscribes to: /1.0/events (lifecycle stream)
  │
  ├──→ Incus API (https://192.168.102.140-142:8443)
  │    All instance/network/storage operations
  │
  └──→ Aether's own REST API (/api/*)
       Auth, clusters, ACLs only
       JWT-authenticated, separate from Incus

CLI (dev-vm-beelink)
  │  Client cert: CN=maarten@dev-vm-beelink
  │
  └──→ Incus API (same endpoints)

Cluster nodes (oc-node-01/02/03)
  │  Client certs: CN=root@oc-node-*
  │
  └──→ Inter-node API (internal cluster operations)

Capture Methods

Primary: Incus Event Stream

The incus monitor command subscribes to the Incus event websocket and captures every API call, operation lifecycle change, and state transition.

# Quick capture (60 seconds)
incusos/helpers/incus-mitm --capture 60

# Long capture with custom output
incusos/helpers/incus-mitm --capture 300 --output /tmp/my-capture.json

# Live streaming (Ctrl+C to stop)
incusos/helpers/incus-mitm --live

# Filter for Aether only
incusos/helpers/incus-mitm --live --filter aether

What gets captured:

  • Every REST API call (method, path, caller certificate)
  • Operation lifecycle (Pending → Running → Success/Failure)
  • Instance lifecycle events (created, started, stopped, deleted, etc.)
  • Cluster-internal operations (node-to-node forwarding)

What doesn't get captured:

  • Request/response bodies (only the path and method)
  • Aether-internal operations (JWT API calls don't go through Incus)
  • Timing/latency data (only timestamps)

Secondary: Aether Go Log

Aether has a live log viewable at /logs/live in the web UI (sidebar → "View Live GO Log"). This shows Aether's perspective: what it decides to do and what errors it encounters.

Access via Playwright (session-authenticated route):

source env
NODE_PATH=~/node_modules node incusos/helpers/aether-browser screenshot /logs/live

Tertiary: mitmproxy (TLS Interception)

For capturing full request/response bodies, a mitmproxy reverse proxy can be deployed. This is complex due to mutual TLS and is only needed if the event stream doesn't provide enough detail.

Not implemented — the event stream proved sufficient for understanding the API protocol.

Analyzing Captures

# Full analysis
incusos/helpers/incus-mitm --analyze /tmp/incus-events.json

# Show only Aether's calls
incusos/helpers/incus-mitm --analyze /tmp/incus-events.json --filter aether

# Identify callers by certificate
incusos/helpers/incus-mitm --callers /tmp/incus-events.json

Certificate Fingerprints

Each caller is identified by their TLS client certificate fingerprint. The event stream logs certificate matching, allowing us to identify who makes each API call.

Fingerprint Prefix Subject Role
bb2dc9eac3d3... CN=maarten@dev-vm-beelink CLI client
6cfd2a1949a7... CN=root@oc-server Aether SDN controller
a68291ca3683... CN=root@oc-node-02 Cluster inter-node

Aether's cert (root@oc-server) is the key one — it identifies all operations that Aether initiates against the Incus cluster.

Captured API Patterns

Instance Stop (CLI-initiated)

CLI  → PUT  /1.0/instances/<name>/state          (body: {"action":"stop"})
CLI  → GET  /1.0/operations/<uuid>                (poll for completion)
Incus → operation: Stopping instance [Pending → Running → Success]
Incus → lifecycle: instance-shutdown

Instance Start (CLI-initiated)

CLI  → PUT  /1.0/instances/<name>/state          (body: {"action":"start"})
CLI  → GET  /1.0/operations/<uuid>
Incus → operation: Starting instance [Pending → Running → Success]
Incus → lifecycle: instance-started

Snapshot Create

CLI  → POST /1.0/instances/<name>/snapshots       (body: {"name":"snap-name"})
CLI  → GET  /1.0/operations/<uuid>
Incus → operation: Snapshotting instance [Pending → Running → Success]
Incus → lifecycle: instance-snapshot-created
Aether → GET /1.0                                  (reacts to lifecycle event)
Aether → GET /1.0/instances/<snap-name>?project=default&recursion=1

Snapshot Delete

CLI  → DELETE /1.0/instances/<name>/snapshots/<snap-name>
CLI  → GET   /1.0/operations/<uuid>
Incus → operation: Deleting snapshot [Pending → Running → Success]
Incus → lifecycle: instance-snapshot-deleted
Aether → GET /1.0                                  (reacts to lifecycle event)
Aether → GET /1.0/instances/<snap-name>?project=default&recursion=1

Exec Command

CLI  → POST /1.0/instances/<name>/exec            (body: {"command":["hostname"],...})
Node → POST https://<target-node>:8443/1.0/instances/<name>/exec  (forwarded)
CLI  → GET  /1.0/operations/<uuid>/websocket?secret=<stdin>
CLI  → GET  /1.0/operations/<uuid>/websocket?secret=<stdout>
CLI  → GET  /1.0/operations/<uuid>/websocket?secret=<stderr>
CLI  → GET  /1.0/operations/<uuid>/websocket?secret=<control>
CLI  → GET  /1.0/operations/<uuid>                (poll completion)
Incus → operation: Executing command [Pending → Running → Success]
Incus → lifecycle: instance-exec

Note: exec uses 4 websocket connections (stdin, stdout, stderr, control). The operation is forwarded to the target node where the instance runs.

Network List

CLI  → GET /1.0/networks?filter=&recursion=1

Instance Info

CLI  → GET /1.0/instances/<name>?recursion=1

Key Findings

1. Aether Subscribes to Lifecycle Events

Aether maintains a persistent connection to the Incus event stream. When any lifecycle event occurs (instance created, snapshot deleted, etc.), Aether immediately queries the affected resource. This is how Aether keeps its UI in sync with the cluster state.

Pattern: lifecycle event → Aether GET /1.0 → Aether GET /1.0/instances/<name>

2. Aether Uses Client Certificate Auth

Aether authenticates to Incus using a TLS client certificate (CN=root@oc-server), not API tokens. This certificate is trusted by all cluster members. The certificate is stored in the Aether VM (likely at /root/.config/incus/client.crt).

3. Operations Are Forwarded

When an operation targets an instance on a different node, the receiving node forwards the request internally. This is visible in the event stream as a POST https://<target-node>:8443/... call from an "internal" caller.

4. Every CLI Command Starts with GET /1.0

The Incus CLI always fetches /1.0 first (server info/capabilities) before making any other call. It also opens an /1.0/events websocket for real-time operation status updates.

5. Aether REST API Is Separate

Aether's own API (/api/auth/token, /api/clusters, /api/clusters/*/rules) is completely separate from the Incus API. These calls go to Aether's HTTPS server (port 8443) and use JWT authentication. They are NOT visible in the Incus event stream.

Current Aether API coverage:

  • POST /api/auth/token — JWT authentication
  • GET /api/clusters — list managed clusters
  • CRUD /api/clusters/{id}/rules — per-cluster ACL rules
  • CRUD /api/global/rules — global ACL rules

All other Aether features (deploys, HAProxy, blueprints, instance management) are UI-only and use session-authenticated routes with CSRF protection. These features call the Incus API directly via Aether's server-side code, visible in the event stream as calls from the root@oc-server certificate.

Reproducing the Capture

# 1. Start capture in one terminal
incusos/helpers/incus-mitm --live

# 2. In another terminal, trigger operations:
incus stop oc-node-01:<instance>
incus start oc-node-01:<instance>
incus snapshot create oc-node-01:<instance> test-snap
incus snapshot delete oc-node-01:<instance> test-snap

# 3. Or trigger via Aether UI:
#    - Deploy a container
#    - Create a blueprint
#    - Manage HAProxy
#    (All will appear in the event stream as Aether API calls)