Add API interception guide, incus-mitm helper, and payload rules
Phase 2: captured Aether↔Incus API traffic using the event stream. Identified callers by TLS certificate fingerprint (CLI vs Aether vs cluster nodes). Documented API patterns for stop/start/snapshot/exec. Key finding: Aether subscribes to lifecycle events and reactively queries affected resources. Deliverables: - notes/api-interception-guide.md: capture setup + all API patterns - incusos/helpers/incus-mitm: --capture/--live/--analyze/--callers - .claude/rules/aether-api-payloads.md: context for AI assistants - .mcp.json: add --ignore-https-errors for Playwright Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
94eb3c7cc4
commit
91a87456a5
|
|
@ -0,0 +1,49 @@
|
||||||
|
# Aether API Payloads Context
|
||||||
|
|
||||||
|
paths:
|
||||||
|
- incus-mitm
|
||||||
|
- api-interception
|
||||||
|
|
||||||
|
## How Aether Communicates with Incus
|
||||||
|
|
||||||
|
Aether (192.168.102.160) authenticates to the Incus cluster using a TLS
|
||||||
|
client certificate with subject `CN=root@oc-server`. Its fingerprint
|
||||||
|
prefix is `6cfd2a1949a7`.
|
||||||
|
|
||||||
|
### Aether's Behavior Pattern
|
||||||
|
|
||||||
|
1. Subscribes to Incus event stream (persistent websocket)
|
||||||
|
2. Reacts to lifecycle events by querying affected resources
|
||||||
|
3. When user triggers UI actions, makes Incus API calls server-side
|
||||||
|
|
||||||
|
### Capturing Events
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Live stream with Aether filter
|
||||||
|
incusos/helpers/incus-mitm --live --filter aether
|
||||||
|
|
||||||
|
# Capture to file
|
||||||
|
incusos/helpers/incus-mitm --capture 120
|
||||||
|
incusos/helpers/incus-mitm --analyze /tmp/incus-events-*.json --filter aether
|
||||||
|
```
|
||||||
|
|
||||||
|
### API Call Patterns
|
||||||
|
|
||||||
|
All Incus operations follow: `request → operation (pending→running→success) → lifecycle event`
|
||||||
|
|
||||||
|
Key patterns:
|
||||||
|
- Stop: `PUT /1.0/instances/<name>/state`
|
||||||
|
- Start: `PUT /1.0/instances/<name>/state`
|
||||||
|
- Snapshot: `POST /1.0/instances/<name>/snapshots`
|
||||||
|
- Delete: `DELETE /1.0/instances/<name>/snapshots/<snap>`
|
||||||
|
- Exec: `POST /1.0/instances/<name>/exec` (4 websockets)
|
||||||
|
|
||||||
|
Aether reacts to lifecycle events with: `GET /1.0 → GET /1.0/instances/<name>?recursion=1`
|
||||||
|
|
||||||
|
### Aether's Own REST API
|
||||||
|
|
||||||
|
Separate from Incus, JWT-authenticated:
|
||||||
|
- `POST /api/auth/token`
|
||||||
|
- `GET /api/clusters`
|
||||||
|
- `CRUD /api/clusters/{id}/rules`
|
||||||
|
- `CRUD /api/global/rules`
|
||||||
|
|
@ -2,7 +2,7 @@
|
||||||
"mcpServers": {
|
"mcpServers": {
|
||||||
"playwright": {
|
"playwright": {
|
||||||
"command": "npx",
|
"command": "npx",
|
||||||
"args": ["@playwright/mcp@latest"]
|
"args": ["@playwright/mcp@latest", "--ignore-https-errors"]
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -32,7 +32,8 @@ incus-contrib/
|
||||||
│ │ ├── proxmox-screenshot # VMID -> PNG console screenshot
|
│ │ ├── proxmox-screenshot # VMID -> PNG console screenshot
|
||||||
│ │ ├── proxmox-api # Authenticated API calls (handles ! in token)
|
│ │ ├── proxmox-api # Authenticated API calls (handles ! in token)
|
||||||
│ │ ├── aether-browser # Playwright browser automation for Aether web UI
|
│ │ ├── aether-browser # Playwright browser automation for Aether web UI
|
||||||
│ │ └── ovn-inspect # OVN/OVS topology inspection (--nb/--sb/--ovs/--trace)
|
│ │ ├── ovn-inspect # OVN/OVS topology inspection (--nb/--sb/--ovs/--trace)
|
||||||
|
│ │ └── incus-mitm # API traffic capture and analysis (--capture/--live/--analyze)
|
||||||
│ ├── lab-test # Guided lab validation (12 test phases)
|
│ ├── lab-test # Guided lab validation (12 test phases)
|
||||||
│ ├── observe-deploy # Single-VM deploy with console screenshots
|
│ ├── observe-deploy # Single-VM deploy with console screenshots
|
||||||
│ ├── proxmox.yaml # Proxmox connection (gitignored)
|
│ ├── proxmox.yaml # Proxmox connection (gitignored)
|
||||||
|
|
@ -187,6 +188,7 @@ which files are being edited:
|
||||||
| `awx-integration.md` | ansible/, deploy-awx, awx-manifests, AWX guide |
|
| `awx-integration.md` | ansible/, deploy-awx, awx-manifests, AWX guide |
|
||||||
| `haproxy-lb.md` | deploy-haproxy, HAProxy guide |
|
| `haproxy-lb.md` | deploy-haproxy, HAProxy guide |
|
||||||
| `ovn-internals.md` | ovn-inspect, ovn-deep-dive |
|
| `ovn-internals.md` | ovn-inspect, ovn-deep-dive |
|
||||||
|
| `aether-api-payloads.md` | incus-mitm, api-interception |
|
||||||
| `proxmox-ssh-rules.md` | **Always loaded** (no paths filter) |
|
| `proxmox-ssh-rules.md` | **Always loaded** (no paths filter) |
|
||||||
| `lab-infrastructure.md` | incusos-proxmox, lab-test, examples |
|
| `lab-infrastructure.md` | incusos-proxmox, lab-test, examples |
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,397 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
# incus-mitm -- Capture and analyze Incus API traffic via event stream
|
||||||
|
#
|
||||||
|
# Usage: incus-mitm [ACTION] [OPTIONS]
|
||||||
|
#
|
||||||
|
# Actions: --capture, --analyze, --live, --callers
|
||||||
|
# Requires: incus CLI with cluster remote configured, python3
|
||||||
|
#
|
||||||
|
# The primary capture method uses `incus monitor` which captures all
|
||||||
|
# API operations without requiring TLS interception.
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Colors
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
setup_colors() {
|
||||||
|
if [[ "${NO_COLOR:-}" == "1" || "${TERM:-}" == "dumb" ]]; then
|
||||||
|
BOLD="" DIM="" RESET="" CYAN="" GREEN="" YELLOW="" RED="" BLUE=""
|
||||||
|
else
|
||||||
|
BOLD=$'\033[1m' DIM=$'\033[2m' RESET=$'\033[0m'
|
||||||
|
CYAN=$'\033[36m' GREEN=$'\033[32m' YELLOW=$'\033[33m'
|
||||||
|
RED=$'\033[31m' BLUE=$'\033[34m'
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
setup_colors
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Usage
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
usage() {
|
||||||
|
cat <<'EOF'
|
||||||
|
Usage: incus-mitm [ACTION] [OPTIONS]
|
||||||
|
|
||||||
|
Capture and analyze Incus REST API traffic using the event stream.
|
||||||
|
No TLS interception needed — uses `incus monitor` to observe all
|
||||||
|
API calls, operations, and lifecycle events in real time.
|
||||||
|
|
||||||
|
Actions:
|
||||||
|
--capture [SECONDS] Capture events for N seconds (default: 60)
|
||||||
|
--live Stream events in real time (Ctrl+C to stop)
|
||||||
|
--analyze FILE Analyze a previously captured JSON file
|
||||||
|
--callers FILE Show unique API callers by certificate
|
||||||
|
|
||||||
|
Options:
|
||||||
|
--remote NAME Incus remote name (default: oc-node-01)
|
||||||
|
--output FILE Output file for capture (default: /tmp/incus-events-<timestamp>.json)
|
||||||
|
--filter CALLER Only show events from specific caller (aether, cli, node)
|
||||||
|
--dry-run Show commands that would be executed
|
||||||
|
--help, -h Show this help
|
||||||
|
|
||||||
|
Examples:
|
||||||
|
incus-mitm --capture 120 # Capture 2 minutes of events
|
||||||
|
incus-mitm --capture 30 --output events.json
|
||||||
|
incus-mitm --live # Stream in real time
|
||||||
|
incus-mitm --analyze /tmp/incus-events.json
|
||||||
|
incus-mitm --callers /tmp/incus-events.json
|
||||||
|
incus-mitm --analyze events.json --filter aether
|
||||||
|
|
||||||
|
Certificate Fingerprint Reference:
|
||||||
|
Capture events first, then use --callers to identify who is who.
|
||||||
|
Common patterns:
|
||||||
|
CN=maarten@* → CLI client
|
||||||
|
CN=root@oc-* → Aether SDN controller
|
||||||
|
CN=root@oc-node-* → Cluster inter-node traffic
|
||||||
|
EOF
|
||||||
|
exit 0
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Defaults and argument parsing
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
ACTION=""
|
||||||
|
REMOTE="oc-node-01"
|
||||||
|
OUTPUT=""
|
||||||
|
CAPTURE_SECONDS=60
|
||||||
|
FILTER=""
|
||||||
|
DRY_RUN=false
|
||||||
|
ANALYZE_FILE=""
|
||||||
|
|
||||||
|
while [[ $# -gt 0 ]]; do
|
||||||
|
case "$1" in
|
||||||
|
--capture)
|
||||||
|
ACTION="capture"
|
||||||
|
if [[ "${2:-}" =~ ^[0-9]+$ ]]; then
|
||||||
|
CAPTURE_SECONDS="$2"
|
||||||
|
shift
|
||||||
|
fi
|
||||||
|
shift
|
||||||
|
;;
|
||||||
|
--live) ACTION="live"; shift ;;
|
||||||
|
--analyze)
|
||||||
|
ACTION="analyze"
|
||||||
|
ANALYZE_FILE="${2:?--analyze requires a file}"
|
||||||
|
shift 2
|
||||||
|
;;
|
||||||
|
--callers)
|
||||||
|
ACTION="callers"
|
||||||
|
ANALYZE_FILE="${2:?--callers requires a file}"
|
||||||
|
shift 2
|
||||||
|
;;
|
||||||
|
--remote) REMOTE="${2:?--remote requires a name}"; shift 2 ;;
|
||||||
|
--output) OUTPUT="${2:?--output requires a path}"; shift 2 ;;
|
||||||
|
--filter) FILTER="${2:?--filter requires a caller type}"; shift 2 ;;
|
||||||
|
--dry-run) DRY_RUN=true; shift ;;
|
||||||
|
-h|--help) usage ;;
|
||||||
|
*)
|
||||||
|
echo "Error: unknown option: $1" >&2
|
||||||
|
echo "Run 'incus-mitm --help' for usage." >&2
|
||||||
|
exit 1
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
|
if [[ -z "$ACTION" ]]; then
|
||||||
|
usage
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Capture
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
do_capture() {
|
||||||
|
local output="${OUTPUT:-/tmp/incus-events-$(date +%Y%m%d-%H%M%S).json}"
|
||||||
|
|
||||||
|
if [[ "$DRY_RUN" == true ]]; then
|
||||||
|
echo "${DIM}[dry-run] timeout ${CAPTURE_SECONDS} incus monitor ${REMOTE}: --format=json > ${output}${RESET}"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "${BOLD}Capturing Incus events for ${CAPTURE_SECONDS}s...${RESET}"
|
||||||
|
echo "${DIM}Output: ${output}${RESET}"
|
||||||
|
echo "${DIM}Trigger operations in Aether or via CLI to generate events.${RESET}"
|
||||||
|
echo
|
||||||
|
|
||||||
|
timeout "$CAPTURE_SECONDS" incus monitor "${REMOTE}:" --format=json > "$output" 2>&1 || true
|
||||||
|
|
||||||
|
local count
|
||||||
|
count=$(wc -l < "$output")
|
||||||
|
echo
|
||||||
|
echo "${GREEN}Captured ${count} events → ${output}${RESET}"
|
||||||
|
echo "Run: ${BOLD}incus-mitm --analyze ${output}${RESET}"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Live stream
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
do_live() {
|
||||||
|
if [[ "$DRY_RUN" == true ]]; then
|
||||||
|
echo "${DIM}[dry-run] incus monitor ${REMOTE}: --format=json | python3 (live parser)${RESET}"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "${BOLD}Live event stream (Ctrl+C to stop)${RESET}"
|
||||||
|
echo
|
||||||
|
|
||||||
|
incus monitor "${REMOTE}:" --format=json 2>&1 | python3 -u -c "
|
||||||
|
import sys, json
|
||||||
|
|
||||||
|
# Known fingerprint prefixes (populated during session)
|
||||||
|
known_certs = {}
|
||||||
|
|
||||||
|
for line in sys.stdin:
|
||||||
|
line = line.strip()
|
||||||
|
if not line:
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
e = json.loads(line)
|
||||||
|
except:
|
||||||
|
continue
|
||||||
|
|
||||||
|
ts = e.get('timestamp', '')[11:19]
|
||||||
|
etype = e.get('type', '')
|
||||||
|
meta = e.get('metadata', {})
|
||||||
|
|
||||||
|
# Track cert fingerprints
|
||||||
|
if etype == 'logging' and 'Matched trusted cert' in meta.get('message', ''):
|
||||||
|
ctx = meta.get('context', {})
|
||||||
|
fp = ctx.get('fingerprint', '')
|
||||||
|
subj = ctx.get('subject', '')
|
||||||
|
if fp and subj:
|
||||||
|
known_certs[fp[:20]] = subj
|
||||||
|
|
||||||
|
# Show API calls
|
||||||
|
if etype == 'logging' and 'method' in meta.get('context', {}):
|
||||||
|
ctx = meta['context']
|
||||||
|
fp = ctx.get('username', '')[:20]
|
||||||
|
caller = known_certs.get(fp, fp[:12] + '...')
|
||||||
|
if 'maarten' in caller:
|
||||||
|
caller = 'CLI'
|
||||||
|
elif 'oc-server' in caller:
|
||||||
|
caller = 'Aether'
|
||||||
|
elif 'oc-node' in caller:
|
||||||
|
caller = caller.split('root@')[1].split(',')[0] if 'root@' in caller else 'node'
|
||||||
|
|
||||||
|
filter_val = '${FILTER}'.lower()
|
||||||
|
if filter_val:
|
||||||
|
if filter_val == 'aether' and 'Aether' not in caller:
|
||||||
|
continue
|
||||||
|
elif filter_val == 'cli' and 'CLI' not in caller:
|
||||||
|
continue
|
||||||
|
elif filter_val == 'node' and 'oc-node' not in caller:
|
||||||
|
continue
|
||||||
|
|
||||||
|
print(f' \033[2m{ts}\033[0m {ctx[\"method\"]:6s} {ctx[\"url\"]:70s} \033[36m[{caller}]\033[0m', flush=True)
|
||||||
|
|
||||||
|
# Show lifecycle events
|
||||||
|
elif etype == 'lifecycle':
|
||||||
|
action = meta.get('action', '')
|
||||||
|
source = (meta.get('source') or '').split('/')[-1]
|
||||||
|
print(f' \033[2m{ts}\033[0m \033[33m{action:30s}\033[0m {source}', flush=True)
|
||||||
|
|
||||||
|
# Show operation status changes (only success/failure)
|
||||||
|
elif etype == 'operation':
|
||||||
|
status = meta.get('status', '')
|
||||||
|
if status in ('Success', 'Failure'):
|
||||||
|
desc = meta.get('description', '')
|
||||||
|
resources = []
|
||||||
|
for k, v in (meta.get('resources') or {}).items():
|
||||||
|
for item in v:
|
||||||
|
resources.append(item.split('/')[-1])
|
||||||
|
res = ', '.join(resources)
|
||||||
|
color = '\033[32m' if status == 'Success' else '\033[31m'
|
||||||
|
print(f' \033[2m{ts}\033[0m {color}[{status:7s}]\033[0m {desc:30s} {res}', flush=True)
|
||||||
|
"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Analyze
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
do_analyze() {
|
||||||
|
if [[ ! -f "$ANALYZE_FILE" ]]; then
|
||||||
|
echo "Error: file not found: $ANALYZE_FILE" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
python3 -c "
|
||||||
|
import json, sys
|
||||||
|
from collections import defaultdict
|
||||||
|
|
||||||
|
events = []
|
||||||
|
with open('${ANALYZE_FILE}') as f:
|
||||||
|
for line in f:
|
||||||
|
line = line.strip()
|
||||||
|
if not line: continue
|
||||||
|
try: events.append(json.loads(line))
|
||||||
|
except: pass
|
||||||
|
|
||||||
|
# Build cert map
|
||||||
|
cert_map = {}
|
||||||
|
for e in events:
|
||||||
|
if e.get('type') == 'logging' and 'Matched trusted cert' in e.get('metadata', {}).get('message', ''):
|
||||||
|
ctx = e['metadata']['context']
|
||||||
|
fp = ctx.get('fingerprint', '')
|
||||||
|
subj = ctx.get('subject', '')
|
||||||
|
if fp and subj:
|
||||||
|
cert_map[fp[:20]] = subj
|
||||||
|
|
||||||
|
def caller_name(fp):
|
||||||
|
subj = cert_map.get(fp[:20], '')
|
||||||
|
if 'maarten' in subj: return 'CLI'
|
||||||
|
if 'oc-server' in subj: return 'Aether'
|
||||||
|
if 'oc-node' in subj:
|
||||||
|
return subj.split('root@')[1].split(',')[0] if 'root@' in subj else 'node'
|
||||||
|
return fp[:12] + '...'
|
||||||
|
|
||||||
|
filter_val = '${FILTER}'.lower()
|
||||||
|
|
||||||
|
print('\033[1m\033[36m=== API Calls ===\033[0m')
|
||||||
|
api_calls = []
|
||||||
|
for e in events:
|
||||||
|
if e.get('type') == 'logging' and 'method' in e.get('metadata', {}).get('context', {}):
|
||||||
|
ctx = e['metadata']['context']
|
||||||
|
caller = caller_name(ctx.get('username', ''))
|
||||||
|
if filter_val:
|
||||||
|
if filter_val == 'aether' and caller != 'Aether': continue
|
||||||
|
elif filter_val == 'cli' and caller != 'CLI': continue
|
||||||
|
ts = e['timestamp'][11:19]
|
||||||
|
api_calls.append((ts, ctx['method'], ctx['url'], caller))
|
||||||
|
print(f\" {ts} {ctx['method']:6s} {ctx['url']:70s} [{caller}]\")
|
||||||
|
|
||||||
|
print()
|
||||||
|
print('\033[1m\033[36m=== Operations ===\033[0m')
|
||||||
|
seen = set()
|
||||||
|
for e in events:
|
||||||
|
if e.get('type') == 'operation':
|
||||||
|
meta = e.get('metadata', {})
|
||||||
|
key = f\"{meta.get('id')}_{meta.get('status')}\"
|
||||||
|
if key in seen: continue
|
||||||
|
seen.add(key)
|
||||||
|
status = meta.get('status', '')
|
||||||
|
if status not in ('Success', 'Failure', 'Cancelled'): continue
|
||||||
|
resources = []
|
||||||
|
for k, v in (meta.get('resources') or {}).items():
|
||||||
|
for item in v: resources.append(item.split('/')[-1])
|
||||||
|
ts = e['timestamp'][11:19]
|
||||||
|
color = '\033[32m' if status == 'Success' else '\033[31m'
|
||||||
|
print(f\" {ts} {color}[{status:7s}]\033[0m {meta.get('description'):30s} {', '.join(resources)}\")
|
||||||
|
|
||||||
|
print()
|
||||||
|
print('\033[1m\033[36m=== Lifecycle Events ===\033[0m')
|
||||||
|
for e in events:
|
||||||
|
if e.get('type') == 'lifecycle':
|
||||||
|
meta = e.get('metadata', {})
|
||||||
|
ts = e['timestamp'][11:19]
|
||||||
|
source = (meta.get('source') or '').split('/')[-1]
|
||||||
|
print(f\" {ts} {meta.get('action'):30s} {source}\")
|
||||||
|
|
||||||
|
print()
|
||||||
|
print('\033[1m\033[36m=== Summary ===\033[0m')
|
||||||
|
print(f' Total events: {len(events)}')
|
||||||
|
print(f' API calls: {len(api_calls)}')
|
||||||
|
callers = defaultdict(int)
|
||||||
|
methods = defaultdict(int)
|
||||||
|
for _, method, url, caller in api_calls:
|
||||||
|
callers[caller] += 1
|
||||||
|
methods[method] += 1
|
||||||
|
print(f' By caller:')
|
||||||
|
for c, n in sorted(callers.items(), key=lambda x: -x[1]):
|
||||||
|
print(f' {c}: {n}')
|
||||||
|
print(f' By method:')
|
||||||
|
for m, n in sorted(methods.items(), key=lambda x: -x[1]):
|
||||||
|
print(f' {m}: {n}')
|
||||||
|
"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Callers
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
do_callers() {
|
||||||
|
if [[ ! -f "$ANALYZE_FILE" ]]; then
|
||||||
|
echo "Error: file not found: $ANALYZE_FILE" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
python3 -c "
|
||||||
|
import json
|
||||||
|
from collections import defaultdict
|
||||||
|
|
||||||
|
events = []
|
||||||
|
with open('${ANALYZE_FILE}') as f:
|
||||||
|
for line in f:
|
||||||
|
line = line.strip()
|
||||||
|
if not line: continue
|
||||||
|
try: events.append(json.loads(line))
|
||||||
|
except: pass
|
||||||
|
|
||||||
|
cert_map = {}
|
||||||
|
call_counts = defaultdict(int)
|
||||||
|
|
||||||
|
for e in events:
|
||||||
|
if e.get('type') == 'logging':
|
||||||
|
meta = e.get('metadata', {})
|
||||||
|
ctx = meta.get('context', {})
|
||||||
|
if 'Matched trusted cert' in meta.get('message', ''):
|
||||||
|
fp = ctx.get('fingerprint', '')
|
||||||
|
subj = ctx.get('subject', '')
|
||||||
|
if fp and subj:
|
||||||
|
cert_map[fp] = subj
|
||||||
|
if 'method' in ctx:
|
||||||
|
call_counts[ctx.get('username', '')] += 1
|
||||||
|
|
||||||
|
print('\033[1m\033[36m=== Certificate Fingerprints ===\033[0m')
|
||||||
|
for fp, subj in sorted(cert_map.items(), key=lambda x: x[1]):
|
||||||
|
calls = call_counts.get(fp, 0)
|
||||||
|
if 'maarten' in subj:
|
||||||
|
role = 'CLI client'
|
||||||
|
elif 'oc-server' in subj:
|
||||||
|
role = 'Aether SDN controller'
|
||||||
|
elif 'oc-node' in subj:
|
||||||
|
role = 'Cluster node'
|
||||||
|
else:
|
||||||
|
role = 'Unknown'
|
||||||
|
print(f' {fp[:40]}...')
|
||||||
|
print(f' Subject: {subj}')
|
||||||
|
print(f' Role: {role}')
|
||||||
|
print(f' Calls: {calls}')
|
||||||
|
print()
|
||||||
|
"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Main
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
case "$ACTION" in
|
||||||
|
capture) do_capture ;;
|
||||||
|
live) do_live ;;
|
||||||
|
analyze) do_analyze ;;
|
||||||
|
callers) do_callers ;;
|
||||||
|
esac
|
||||||
|
|
@ -0,0 +1,249 @@
|
||||||
|
# API Interception Guide: Observing Aether ↔ Incus Communication
|
||||||
|
|
||||||
|
How to capture, analyze, and understand the REST API calls between
|
||||||
|
Aether (SDN controller) and the Incus cluster at the wire level.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
Aether (192.168.102.160)
|
||||||
|
│ Client cert: CN=root@oc-server
|
||||||
|
│ Subscribes to: /1.0/events (lifecycle stream)
|
||||||
|
│
|
||||||
|
├──→ Incus API (https://192.168.102.140-142:8443)
|
||||||
|
│ All instance/network/storage operations
|
||||||
|
│
|
||||||
|
└──→ Aether's own REST API (/api/*)
|
||||||
|
Auth, clusters, ACLs only
|
||||||
|
JWT-authenticated, separate from Incus
|
||||||
|
|
||||||
|
CLI (dev-vm-beelink)
|
||||||
|
│ Client cert: CN=maarten@dev-vm-beelink
|
||||||
|
│
|
||||||
|
└──→ Incus API (same endpoints)
|
||||||
|
|
||||||
|
Cluster nodes (oc-node-01/02/03)
|
||||||
|
│ Client certs: CN=root@oc-node-*
|
||||||
|
│
|
||||||
|
└──→ Inter-node API (internal cluster operations)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Capture Methods
|
||||||
|
|
||||||
|
### Primary: Incus Event Stream
|
||||||
|
|
||||||
|
The `incus monitor` command subscribes to the Incus event websocket and
|
||||||
|
captures every API call, operation lifecycle change, and state transition.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Quick capture (60 seconds)
|
||||||
|
incusos/helpers/incus-mitm --capture 60
|
||||||
|
|
||||||
|
# Long capture with custom output
|
||||||
|
incusos/helpers/incus-mitm --capture 300 --output /tmp/my-capture.json
|
||||||
|
|
||||||
|
# Live streaming (Ctrl+C to stop)
|
||||||
|
incusos/helpers/incus-mitm --live
|
||||||
|
|
||||||
|
# Filter for Aether only
|
||||||
|
incusos/helpers/incus-mitm --live --filter aether
|
||||||
|
```
|
||||||
|
|
||||||
|
**What gets captured:**
|
||||||
|
- Every REST API call (method, path, caller certificate)
|
||||||
|
- Operation lifecycle (Pending → Running → Success/Failure)
|
||||||
|
- Instance lifecycle events (created, started, stopped, deleted, etc.)
|
||||||
|
- Cluster-internal operations (node-to-node forwarding)
|
||||||
|
|
||||||
|
**What doesn't get captured:**
|
||||||
|
- Request/response bodies (only the path and method)
|
||||||
|
- Aether-internal operations (JWT API calls don't go through Incus)
|
||||||
|
- Timing/latency data (only timestamps)
|
||||||
|
|
||||||
|
### Secondary: Aether Go Log
|
||||||
|
|
||||||
|
Aether has a live log viewable at `/logs/live` in the web UI (sidebar →
|
||||||
|
"View Live GO Log"). This shows Aether's perspective: what it decides to
|
||||||
|
do and what errors it encounters.
|
||||||
|
|
||||||
|
Access via Playwright (session-authenticated route):
|
||||||
|
```bash
|
||||||
|
source env
|
||||||
|
NODE_PATH=~/node_modules node incusos/helpers/aether-browser screenshot /logs/live
|
||||||
|
```
|
||||||
|
|
||||||
|
### Tertiary: mitmproxy (TLS Interception)
|
||||||
|
|
||||||
|
For capturing full request/response bodies, a mitmproxy reverse proxy
|
||||||
|
can be deployed. This is complex due to mutual TLS and is only needed
|
||||||
|
if the event stream doesn't provide enough detail.
|
||||||
|
|
||||||
|
**Not implemented** — the event stream proved sufficient for understanding
|
||||||
|
the API protocol.
|
||||||
|
|
||||||
|
## Analyzing Captures
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Full analysis
|
||||||
|
incusos/helpers/incus-mitm --analyze /tmp/incus-events.json
|
||||||
|
|
||||||
|
# Show only Aether's calls
|
||||||
|
incusos/helpers/incus-mitm --analyze /tmp/incus-events.json --filter aether
|
||||||
|
|
||||||
|
# Identify callers by certificate
|
||||||
|
incusos/helpers/incus-mitm --callers /tmp/incus-events.json
|
||||||
|
```
|
||||||
|
|
||||||
|
## Certificate Fingerprints
|
||||||
|
|
||||||
|
Each caller is identified by their TLS client certificate fingerprint.
|
||||||
|
The event stream logs certificate matching, allowing us to identify who
|
||||||
|
makes each API call.
|
||||||
|
|
||||||
|
| Fingerprint Prefix | Subject | Role |
|
||||||
|
|---|---|---|
|
||||||
|
| `bb2dc9eac3d3...` | CN=maarten@dev-vm-beelink | CLI client |
|
||||||
|
| `6cfd2a1949a7...` | CN=root@oc-server | Aether SDN controller |
|
||||||
|
| `a68291ca3683...` | CN=root@oc-node-02 | Cluster inter-node |
|
||||||
|
|
||||||
|
Aether's cert (`root@oc-server`) is the key one — it identifies all
|
||||||
|
operations that Aether initiates against the Incus cluster.
|
||||||
|
|
||||||
|
## Captured API Patterns
|
||||||
|
|
||||||
|
### Instance Stop (CLI-initiated)
|
||||||
|
|
||||||
|
```
|
||||||
|
CLI → PUT /1.0/instances/<name>/state (body: {"action":"stop"})
|
||||||
|
CLI → GET /1.0/operations/<uuid> (poll for completion)
|
||||||
|
Incus → operation: Stopping instance [Pending → Running → Success]
|
||||||
|
Incus → lifecycle: instance-shutdown
|
||||||
|
```
|
||||||
|
|
||||||
|
### Instance Start (CLI-initiated)
|
||||||
|
|
||||||
|
```
|
||||||
|
CLI → PUT /1.0/instances/<name>/state (body: {"action":"start"})
|
||||||
|
CLI → GET /1.0/operations/<uuid>
|
||||||
|
Incus → operation: Starting instance [Pending → Running → Success]
|
||||||
|
Incus → lifecycle: instance-started
|
||||||
|
```
|
||||||
|
|
||||||
|
### Snapshot Create
|
||||||
|
|
||||||
|
```
|
||||||
|
CLI → POST /1.0/instances/<name>/snapshots (body: {"name":"snap-name"})
|
||||||
|
CLI → GET /1.0/operations/<uuid>
|
||||||
|
Incus → operation: Snapshotting instance [Pending → Running → Success]
|
||||||
|
Incus → lifecycle: instance-snapshot-created
|
||||||
|
Aether → GET /1.0 (reacts to lifecycle event)
|
||||||
|
Aether → GET /1.0/instances/<snap-name>?project=default&recursion=1
|
||||||
|
```
|
||||||
|
|
||||||
|
### Snapshot Delete
|
||||||
|
|
||||||
|
```
|
||||||
|
CLI → DELETE /1.0/instances/<name>/snapshots/<snap-name>
|
||||||
|
CLI → GET /1.0/operations/<uuid>
|
||||||
|
Incus → operation: Deleting snapshot [Pending → Running → Success]
|
||||||
|
Incus → lifecycle: instance-snapshot-deleted
|
||||||
|
Aether → GET /1.0 (reacts to lifecycle event)
|
||||||
|
Aether → GET /1.0/instances/<snap-name>?project=default&recursion=1
|
||||||
|
```
|
||||||
|
|
||||||
|
### Exec Command
|
||||||
|
|
||||||
|
```
|
||||||
|
CLI → POST /1.0/instances/<name>/exec (body: {"command":["hostname"],...})
|
||||||
|
Node → POST https://<target-node>:8443/1.0/instances/<name>/exec (forwarded)
|
||||||
|
CLI → GET /1.0/operations/<uuid>/websocket?secret=<stdin>
|
||||||
|
CLI → GET /1.0/operations/<uuid>/websocket?secret=<stdout>
|
||||||
|
CLI → GET /1.0/operations/<uuid>/websocket?secret=<stderr>
|
||||||
|
CLI → GET /1.0/operations/<uuid>/websocket?secret=<control>
|
||||||
|
CLI → GET /1.0/operations/<uuid> (poll completion)
|
||||||
|
Incus → operation: Executing command [Pending → Running → Success]
|
||||||
|
Incus → lifecycle: instance-exec
|
||||||
|
```
|
||||||
|
|
||||||
|
Note: exec uses 4 websocket connections (stdin, stdout, stderr, control).
|
||||||
|
The operation is forwarded to the target node where the instance runs.
|
||||||
|
|
||||||
|
### Network List
|
||||||
|
|
||||||
|
```
|
||||||
|
CLI → GET /1.0/networks?filter=&recursion=1
|
||||||
|
```
|
||||||
|
|
||||||
|
### Instance Info
|
||||||
|
|
||||||
|
```
|
||||||
|
CLI → GET /1.0/instances/<name>?recursion=1
|
||||||
|
```
|
||||||
|
|
||||||
|
## Key Findings
|
||||||
|
|
||||||
|
### 1. Aether Subscribes to Lifecycle Events
|
||||||
|
|
||||||
|
Aether maintains a persistent connection to the Incus event stream.
|
||||||
|
When any lifecycle event occurs (instance created, snapshot deleted, etc.),
|
||||||
|
Aether immediately queries the affected resource. This is how Aether
|
||||||
|
keeps its UI in sync with the cluster state.
|
||||||
|
|
||||||
|
Pattern: `lifecycle event → Aether GET /1.0 → Aether GET /1.0/instances/<name>`
|
||||||
|
|
||||||
|
### 2. Aether Uses Client Certificate Auth
|
||||||
|
|
||||||
|
Aether authenticates to Incus using a TLS client certificate
|
||||||
|
(`CN=root@oc-server`), not API tokens. This certificate is trusted by
|
||||||
|
all cluster members. The certificate is stored in the Aether VM
|
||||||
|
(likely at `/root/.config/incus/client.crt`).
|
||||||
|
|
||||||
|
### 3. Operations Are Forwarded
|
||||||
|
|
||||||
|
When an operation targets an instance on a different node, the receiving
|
||||||
|
node forwards the request internally. This is visible in the event stream
|
||||||
|
as a `POST https://<target-node>:8443/...` call from an "internal" caller.
|
||||||
|
|
||||||
|
### 4. Every CLI Command Starts with GET /1.0
|
||||||
|
|
||||||
|
The Incus CLI always fetches `/1.0` first (server info/capabilities)
|
||||||
|
before making any other call. It also opens an `/1.0/events` websocket
|
||||||
|
for real-time operation status updates.
|
||||||
|
|
||||||
|
### 5. Aether REST API Is Separate
|
||||||
|
|
||||||
|
Aether's own API (`/api/auth/token`, `/api/clusters`, `/api/clusters/*/rules`)
|
||||||
|
is completely separate from the Incus API. These calls go to Aether's
|
||||||
|
HTTPS server (port 8443) and use JWT authentication. They are NOT visible
|
||||||
|
in the Incus event stream.
|
||||||
|
|
||||||
|
Current Aether API coverage:
|
||||||
|
- `POST /api/auth/token` — JWT authentication
|
||||||
|
- `GET /api/clusters` — list managed clusters
|
||||||
|
- `CRUD /api/clusters/{id}/rules` — per-cluster ACL rules
|
||||||
|
- `CRUD /api/global/rules` — global ACL rules
|
||||||
|
|
||||||
|
All other Aether features (deploys, HAProxy, blueprints, instance
|
||||||
|
management) are UI-only and use session-authenticated routes with CSRF
|
||||||
|
protection. These features call the Incus API directly via Aether's
|
||||||
|
server-side code, visible in the event stream as calls from the
|
||||||
|
`root@oc-server` certificate.
|
||||||
|
|
||||||
|
## Reproducing the Capture
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. Start capture in one terminal
|
||||||
|
incusos/helpers/incus-mitm --live
|
||||||
|
|
||||||
|
# 2. In another terminal, trigger operations:
|
||||||
|
incus stop oc-node-01:<instance>
|
||||||
|
incus start oc-node-01:<instance>
|
||||||
|
incus snapshot create oc-node-01:<instance> test-snap
|
||||||
|
incus snapshot delete oc-node-01:<instance> test-snap
|
||||||
|
|
||||||
|
# 3. Or trigger via Aether UI:
|
||||||
|
# - Deploy a container
|
||||||
|
# - Create a blueprint
|
||||||
|
# - Manage HAProxy
|
||||||
|
# (All will appear in the event stream as Aether API calls)
|
||||||
|
```
|
||||||
Loading…
Reference in New Issue