maarten
/
incus-contrib
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Convert all ASCII diagrams to mermaid with consistent styling 

Replace ASCII art diagrams in 9 guide files with mermaid flowcharts
using a unified style system:

- 5 semantic colors (Okabe-Ito derived, color-blind safe):
  teal (nodes), sky blue (instances), blue (network),
  amber (load balancers), mauve (management)
- classDef for reusable styles, tinted subgraph fills
- Max 2 lines per node, specs in companion tables
- <br/> for line breaks (not \n, which Gitea renders literally)
- Solid arrows for data flow, dashed for control plane

Files converted:
- haproxy-guide.md (1 diagram)
- observability-guide.md (1 diagram)
- api-interception-guide.md (1 diagram)
- incusos-break-fix.md (3 diagrams: partitions, boot chain, updates)
- shared-storage-guide.md (3 diagrams: iSCSI arch, topology, multi-host)
- networking-guide.md (2 diagrams: bridge, OVN topology)
- operations-center-guide.md (1 diagram)
- ovn-deep-dive.md (3 diagrams: lab, bridges, Geneve mesh)
- production-lab-guide.md (1 diagram)
- mermaid-test.md updated as style reference with <br/> fix

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Maarten 2026-02-24 17:31:29 +01:00
parent 6293f7ab79
commit 671d79b77f
10 changed files with 439 additions and 339 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

44
notes/api-interception-guide.md
View File

@ -5,27 +5,33 @@ Aether (SDN controller) and the Incus cluster at the wire level.
## Architecture ## Architecture
``` ```mermaid
Aether (192.168.102.160) flowchart LR
│ Client cert: CN=root@oc-server subgraph callers["API Callers"]
│ Subscribes to: /1.0/events (lifecycle stream) aether["Aether<br/>192.168.102.160<br/>CN=root@oc-server"]
│ cli["CLI<br/>dev-vm-beelink<br/>CN=maarten@dev-vm-beelink"]
├──→ Incus API (https://192.168.102.140-142:8443) nodes["Cluster Nodes<br/>oc-node-01/02/03<br/>CN=root@oc-node-*"]
│ All instance/network/storage operations end
│
└──→ Aether's own REST API (/api/*)
Auth, clusters, ACLs only
JWT-authenticated, separate from Incus
CLI (dev-vm-beelink) incus["Incus API<br/>:8443 on .140-.142"]
│ Client cert: CN=maarten@dev-vm-beelink aether_api["Aether REST API<br/>/api/* (JWT auth)"]
│
└──→ Incus API (same endpoints)
Cluster nodes (oc-node-01/02/03) aether -->|"client cert<br/>+ /1.0/events"| incus
│ Client certs: CN=root@oc-node-* aether --> aether_api
│ cli -->|"client cert"| incus
└──→ Inter-node API (internal cluster operations) nodes -->|"inter-node"| incus
classDef mgmt fill:#CC79A7,color:#fff,stroke:#a36088
classDef instance fill:#56B4E9,color:#fff,stroke:#3a8fbf
classDef network fill:#0072B2,color:#fff,stroke:#005a8e
classDef node fill:#009E73,color:#fff,stroke:#007a5e
class aether mgmt
class cli instance
class nodes node
class incus,aether_api network
style callers fill:#f5f5f5,stroke:#999
``` ```
## Capture Methods ## Capture Methods

56
notes/haproxy-guide.md
View File

@ -10,37 +10,31 @@ creating test backends, configuring services, and verifying load balancing.
## Architecture ## Architecture
``` ```mermaid
Client / LAN traffic flowchart TD
| client(["Client / LAN"])
v vip["VIP 192.168.103.200"]
+---------------------------+ ovnlb("OVN Load Balancer")
| UPLINK VIP Address | ha1["HAProxy 01<br/>10.10.10.50"]
| 192.168.103.200 | ha2["HAProxy 02<br/>10.10.10.51"]
+---------------------------+ ng1["nginx-01 · .60"]
| ng2["nginx-02 · .61"]
v ng3["nginx-03 · .62"]
+---------------------------+
| OVN Load Balancer | client --> vip
| (distributes to HA | vip --> ovnlb
| pair for failover) | ovnlb --> ha1 & ha2
+---------------------------+ ha1 & ha2 --> ng1 & ng2 & ng3
/ \
/ \ classDef external fill:#f5f5f5,color:#333,stroke:#999
v v classDef network fill:#0072B2,color:#fff,stroke:#005a8e
+-----------------+ +-----------------+ classDef lb fill:#E69F00,color:#fff,stroke:#b87d00
| HAProxy 01 | | HAProxy 02 | classDef instance fill:#56B4E9,color:#fff,stroke:#3a8fbf
| 10.10.10.50 | | 10.10.10.51 |
| (container) | | (container) | class client external
+-----------------+ +-----------------+ class vip,ovnlb network
\ | / class ha1,ha2 lb
\ | / class ng1,ng2,ng3 instance
v v v
+----------+ +----------+ +----------+
| nginx-01 | | nginx-02 | | nginx-03 |
| .60 :80 | | .61 :80 | | .62 :80 |
+----------+ +----------+ +----------+
(backend servers on net-prod)
``` ```
### OVN native LB vs HAProxy ### OVN native LB vs HAProxy

98
notes/incusos-break-fix.md
View File

@ -50,16 +50,22 @@ node-01 was already on this version; node-02 and node-03 were on
Each node has a single SCSI disk (QEMU HARDDISK) with this layout: Each node has a single SCSI disk (QEMU HARDDISK) with this layout:
``` ```mermaid
+------------------------------------------------------+ block-beta
| SCSI Disk (64-100 GiB, QEMU HARDDISK) | columns 1
|------------------------------------------------------| disk["SCSI Disk · 64-100 GiB · QEMU HARDDISK"]
| Partition 1 EFI System Partition | p1["Partition 1 — EFI System Partition"]
| Partition A Root filesystem (active or standby) | pA["Partition A — Root filesystem (active or standby)"]
| Partition B Root filesystem (standby or active) | pB["Partition B — Root filesystem (standby or active)"]
| Partition X Swap (encrypted, TPM-unlocked) | pX["Partition X — Swap (encrypted, TPM-unlocked)"]
| Partition 11 ZFS data pool ("local") | p11["Partition 11 — ZFS data pool 'local'"]
+------------------------------------------------------+
style disk fill:#f5f5f5,color:#333,stroke:#999
style p1 fill:#E69F00,color:#fff,stroke:#b87d00
style pA fill:#009E73,color:#fff,stroke:#007a5e
style pB fill:#56B4E9,color:#fff,stroke:#3a8fbf
style pX fill:#CC79A7,color:#fff,stroke:#a36088
style p11 fill:#0072B2,color:#fff,stroke:#005a8e
``` ```
Key observations from the storage API: Key observations from the storage API:
@ -74,15 +80,28 @@ Key observations from the storage API:
The boot security chain, as validated through the API: The boot security chain, as validated through the API:
``` ```mermaid
UEFI firmware flowchart TD
--> Secure Boot certificate validation (PK -> KEK -> db certs) uefi["UEFI Firmware"]
--> Signed kernel + initrd loaded sb["Secure Boot validation<br/>PK → KEK → db certs"]
--> TPM measured boot (PCR measurements recorded) kernel["Signed kernel + initrd loaded"]
--> TPM validates measurements match expected policy tpm["TPM measured boot<br/>PCR measurements recorded"]
--> Root partition decrypted and mounted validate["TPM validates measurements"]
--> Swap partition decrypted and activated root["Root partition decrypted"]
--> System boots into trusted state swap["Swap partition decrypted"]
boot["System boots into trusted state"]
uefi --> sb --> kernel --> tpm --> validate
validate --> root & swap --> boot
classDef security fill:#CC79A7,color:#fff,stroke:#a36088
classDef node fill:#009E73,color:#fff,stroke:#007a5e
classDef network fill:#0072B2,color:#fff,stroke:#005a8e
class uefi,sb,kernel security
class tpm,validate security
class root,swap network
class boot node
``` ```
API-reported security state: API-reported security state:
@ -173,19 +192,34 @@ updates. Observations from querying the update API and live testing:
### Update lifecycle (observed) ### Update lifecycle (observed)
``` ```mermaid
Config: check_frequency set via PUT /os/1.0/system/update flowchart TD
--> Timer fires (or boot-time check runs) config["check_frequency set via<br/>PUT /os/1.0/system/update"]
--> Query stable channel for new version timer["Timer fires<br/>(or boot-time check)"]
--> Download new rootfs to standby partition query["Query stable channel"]
--> os_version_next updated, needs_reboot = true download["Download rootfs to<br/>standby partition"]
--> status = "IncusOS has been updated to version YYYYMMDDHHMI" ready["os_version_next updated<br/>needs_reboot = true"]
--> (if auto_reboot) Reboot automatically
--> (if !auto_reboot) Wait for manual reboot config --> timer --> query --> download --> ready
--> On reboot: boot from new partition
--> TPM re-measures, validates new boot chain ready -->|auto_reboot| reboot
--> If valid: new partition becomes active ready -->|manual| wait["Wait for<br/>manual reboot"]
--> If invalid: rollback to previous partition wait --> reboot["Reboot from<br/>new partition"]
reboot --> tpm["TPM re-measures<br/>validates boot chain"]
tpm -->|valid| active["New partition<br/>becomes active"]
tpm -->|invalid| rollback["Rollback to<br/>previous partition"]
classDef mgmt fill:#CC79A7,color:#fff,stroke:#a36088
classDef node fill:#009E73,color:#fff,stroke:#007a5e
classDef network fill:#0072B2,color:#fff,stroke:#005a8e
classDef lb fill:#E69F00,color:#fff,stroke:#b87d00
class config,timer,query mgmt
class download,ready network
class reboot,tpm,wait lb
class active node
class rollback fill:#D55E00,color:#fff,stroke:#a34a00
``` ```
--- ---

43
notes/mermaid-test.md
View File

@ -1,7 +1,7 @@
# Mermaid Diagram Rendering Test # Mermaid Diagram Rendering Test
Test file to verify Gitea renders mermaid diagrams correctly. Style reference for all mermaid diagrams in this repository.
Delete this file after confirming. See individual guide files for the actual diagrams in use.
--- ---
@ -31,6 +31,7 @@ and minimal text per node.
- Use `classDef` for styling, never per-node `style` statements. - Use `classDef` for styling, never per-node `style` statements.
- No emojis or unicode symbols in labels (renderer compatibility). - No emojis or unicode symbols in labels (renderer compatibility).
- Edge labels: 1-3 words max, only when relationship isn't obvious. - Edge labels: 1-3 words max, only when relationship isn't obvious.
- Use `<br/>` for line breaks in node labels (not `\n` — Gitea renders that literally).
--- ---
@ -43,8 +44,8 @@ flowchart TD
client(["Client / LAN"]) client(["Client / LAN"])
vip["VIP 192.168.103.200"] vip["VIP 192.168.103.200"]
ovnlb("OVN Load Balancer") ovnlb("OVN Load Balancer")
ha1["HAProxy 01\n10.10.10.50"] ha1["HAProxy 01<br/>10.10.10.50"]
ha2["HAProxy 02\n10.10.10.51"] ha2["HAProxy 02<br/>10.10.10.51"]
ng1["nginx-01 · .60"] ng1["nginx-01 · .60"]
ng2["nginx-02 · .61"] ng2["nginx-02 · .61"]
ng3["nginx-03 · .62"] ng3["nginx-03 · .62"]
@ -74,16 +75,16 @@ Original: `notes/networking-guide.md`
```mermaid ```mermaid
flowchart TD flowchart TD
subgraph node1["Node 1 · net-node-01"] subgraph node1["Node 1 · net-node-01"]
c1["c1 · .202"] & c2["c2 · .40"] --- br1("incusbr0\n10.0.0.1/24") c1["c1 · .202"] & c2["c2 · .40"] --- br1("incusbr0<br/>10.0.0.1/24")
br1 --> nat1["NAT · 192.168.1.209"] br1 --> nat1["NAT · 192.168.1.209"]
end end
subgraph node2["Node 2 · net-node-02"] subgraph node2["Node 2 · net-node-02"]
c3["c3 · .52"] --- br2("incusbr0\n10.0.0.1/24") c3["c3 · .52"] --- br2("incusbr0<br/>10.0.0.1/24")
br2 --> nat2["NAT · 192.168.1.150"] br2 --> nat2["NAT · 192.168.1.150"]
end end
nat1 & nat2 --- lan(("LAN\n192.168.1.0/24")) nat1 & nat2 --- lan(("LAN<br/>192.168.1.0/24"))
classDef instance fill:#56B4E9,color:#fff,stroke:#3a8fbf classDef instance fill:#56B4E9,color:#fff,stroke:#3a8fbf
classDef network fill:#0072B2,color:#fff,stroke:#005a8e classDef network fill:#0072B2,color:#fff,stroke:#005a8e
@ -109,22 +110,22 @@ Original: `notes/networking-guide.md`
```mermaid ```mermaid
flowchart TD flowchart TD
subgraph cp["OVN Control Plane"] subgraph cp["OVN Control Plane"]
ovnc["ovn-central\nNB :6641 · SB :6642"] ovnc["ovn-central<br/>NB :6641 · SB :6642"]
end end
subgraph n1["Node 1 · net-node-01"] subgraph n1["Node 1 · net-node-01"]
ctrl1["ovn-controller"] ~~~ ls1 ctrl1["ovn-controller"] ~~~ ls1
c1["c1 · .2"] & c2["c2 · .3"] --- ls1("logical switch\n10.10.10.0/24") c1["c1 · .2"] & c2["c2 · .3"] --- ls1("logical switch<br/>10.10.10.0/24")
end end
subgraph n2["Node 2 · net-node-02"] subgraph n2["Node 2 · net-node-02"]
ctrl2["ovn-controller"] ~~~ ls2 ctrl2["ovn-controller"] ~~~ ls2
c3["c3 · .4"] --- ls2("logical switch\n10.10.10.0/24") c3["c3 · .4"] --- ls2("logical switch<br/>10.10.10.0/24")
end end
subgraph n3["Node 3 · net-node-03"] subgraph n3["Node 3 · net-node-03"]
ctrl3["ovn-controller"] ~~~ ls3 ctrl3["ovn-controller"] ~~~ ls3
c4["c4 · .5"] --- ls3("logical switch\n10.10.10.0/24") c4["c4 · .5"] --- ls3("logical switch<br/>10.10.10.0/24")
end end
ovnc -.-> ctrl1 & ctrl2 & ctrl3 ovnc -.-> ctrl1 & ctrl2 & ctrl3
@ -133,7 +134,7 @@ flowchart TD
ls2 <-->|Geneve| ls3 ls2 <-->|Geneve| ls3
ls1 <-->|Geneve| ls3 ls1 <-->|Geneve| ls3
n1 & n2 & n3 --- lan(("LAN\n192.168.1.0/24")) n1 & n2 & n3 --- lan(("LAN<br/>192.168.1.0/24"))
classDef mgmt fill:#CC79A7,color:#fff,stroke:#a36088 classDef mgmt fill:#CC79A7,color:#fff,stroke:#a36088
classDef instance fill:#56B4E9,color:#fff,stroke:#3a8fbf classDef instance fill:#56B4E9,color:#fff,stroke:#3a8fbf
@ -162,9 +163,9 @@ Original: `notes/ovn-deep-dive.md`
```mermaid ```mermaid
graph LR graph LR
n1(("oc-node-01\n.140")) n1(("oc-node-01<br/>.140"))
n2(("oc-node-02\n.141")) n2(("oc-node-02<br/>.141"))
n3(("oc-node-03\n.142")) n3(("oc-node-03<br/>.142"))
n1 <-->|"Geneve 6081"| n2 n1 <-->|"Geneve 6081"| n2
n2 <-->|"Geneve 6081"| n3 n2 <-->|"Geneve 6081"| n3
@ -224,15 +225,15 @@ Original: `notes/operations-center-guide.md`
flowchart TD flowchart TD
subgraph proxmox["Proxmox VE Host · i9-13900HK · 64 GiB"] subgraph proxmox["Proxmox VE Host · i9-13900HK · 64 GiB"]
subgraph cluster["Incus Cluster · net-prod 10.10.10.0/24"] subgraph cluster["Incus Cluster · net-prod 10.10.10.0/24"]
n1["oc-node-01\nVMID 400 · .140"] n1["oc-node-01<br/>VMID 400 · .140"]
n2["oc-node-02\nVMID 401 · .141"] n2["oc-node-02<br/>VMID 401 · .141"]
n3["oc-node-03\nVMID 402 · .142"] n3["oc-node-03<br/>VMID 402 · .142"]
end end
oc["Operations Center\nVMID 920 · .120"] oc["Operations Center<br/>VMID 920 · .120"]
end end
vmbr0(("VLAN 69\n192.168.100.0/22")) vmbr0(("VLAN 69<br/>192.168.100.0/22"))
ext["OVN external IPs\n192.168.103.200-210"] ext["OVN external IPs<br/>192.168.103.200-210"]
proxmox --- vmbr0 proxmox --- vmbr0
cluster -.->|"external gateway"| ext cluster -.->|"external gateway"| ext

117
notes/networking-guide.md
View File

@ -78,25 +78,30 @@ works without any additional infrastructure.
### Bridge topology ### Bridge topology
``` ```mermaid
Node 1 (net-node-01) Node 2 (net-node-02) flowchart TD
┌─────────────────────┐ ┌─────────────────────┐ subgraph node1["Node 1 · net-node-01"]
│ incusbr0 │ │ incusbr0 │ c1["c1 · .202"] & c2["c2 · .40"] --- br1("incusbr0<br/>10.0.0.1/24")
│ 10.0.0.1/24 │ │ 10.0.0.1/24 │ br1 --> nat1["NAT · 192.168.1.209"]
│ ┌───┐ ┌───┐ │ │ ┌───┐ │ end
│ │c1 │ │c2 │ │ │ │c3 │ │
│ │.202│ │.40│ │ │ │.52│ │ subgraph node2["Node 2 · net-node-02"]
│ └───┘ └───┘ │ │ └───┘ │ c3["c3 · .52"] --- br2("incusbr0<br/>10.0.0.1/24")
│ │ │ │ │ │ │ br2 --> nat2["NAT · 192.168.1.150"]
│ ──┴──────┴── │ │ ──┴── │ end
│ (bridge) │ │ (bridge) │
│ │ │ │ │ │ nat1 & nat2 --- lan(("LAN<br/>192.168.1.0/24"))
│ NAT ──> ens18 │ │ NAT ──> ens18 │
│ 192.168.1.209 │ │ 192.168.1.150 │ classDef instance fill:#56B4E9,color:#fff,stroke:#3a8fbf
└─────────────────────┘ └─────────────────────┘ classDef network fill:#0072B2,color:#fff,stroke:#005a8e
│ │ classDef node fill:#009E73,color:#fff,stroke:#007a5e
────┴──────────────────────────────────┴────
LAN (192.168.1.0/24) class c1,c2,c3 instance
class br1,br2,lan network
class nat1,nat2 node
style node1 fill:#e6f5f0,stroke:#009E73
style node2 fill:#e6f5f0,stroke:#009E73
``` ```
**Key point:** Each node has its own bridge with the same subnet (10.0.0.1/24). **Key point:** Each node has its own bridge with the same subnet (10.0.0.1/24).
@ -193,37 +198,49 @@ Options for deploying the control plane:
### OVN topology ### OVN topology
``` ```mermaid
┌──────────────────────────────────┐ flowchart TD
│ OVN Control Plane │ subgraph cp["OVN Control Plane"]
│ (container: ovn-central) │ ovnc["ovn-central<br/>NB :6641 · SB :6642"]
│ ovn-northd + ovsdb-server │ end
│ NB: tcp:192.168.1.209:6641 │
│ SB: tcp:192.168.1.209:6642 │ subgraph n1["Node 1 · net-node-01"]
└──────────┬───────────────────────┘ ctrl1["ovn-controller"] ~~~ ls1
│ (proxy devices) c1["c1 · .2"] & c2["c2 · .3"] --- ls1("logical switch<br/>10.10.10.0/24")
┌─────────────────────────┼─────────────────────────┐ end
│ │ │
Node 1 (net-node-01) Node 2 (net-node-02) Node 3 (net-node-03) subgraph n2["Node 2 · net-node-02"]
┌───────────────────┐ ┌───────────────────┐ ┌───────────────────┐ ctrl2["ovn-controller"] ~~~ ls2
│ ovn-controller │ │ ovn-controller │ │ ovn-controller │ c3["c3 · .4"] --- ls2("logical switch<br/>10.10.10.0/24")
│ ovs-vswitchd │ │ ovs-vswitchd │ │ ovs-vswitchd │ end
│ │ │ │ │ │
│ ┌───┐ ┌───┐ │ │ ┌───┐ │ │ ┌───┐ │ subgraph n3["Node 3 · net-node-03"]
│ │c1 │ │c2 │ │ │ │c3 │ │ │ │c4 │ │ ctrl3["ovn-controller"] ~~~ ls3
│ │.2 │ │.3 │ │ │ │.4 │ │ │ │.5 │ │ c4["c4 · .5"] --- ls3("logical switch<br/>10.10.10.0/24")
│ └─┬─┘ └─┬─┘ │ │ └─┬─┘ │ │ └─┬─┘ │ end
│ └──┬───┘ │ │ │ │ │ │ │
│ OVN logical │ │ OVN logical │ │ OVN logical │ ovnc -.->|"proxy devices"| ctrl1 & ctrl2 & ctrl3
│ switch │ │ switch │ │ switch │
│ 10.10.10.0/24 │ │ 10.10.10.0/24 │ │ 10.10.10.0/24 │ ls1 <-->|Geneve| ls2
│ │ │ │ │ │ │ │ │ ls2 <-->|Geneve| ls3
│ Geneve tunnel ──┼──┼── Geneve tunnel ──┼──┼── Geneve tunnel │ ls1 <-->|Geneve| ls3
│ 192.168.1.209 │ │ 192.168.1.150 │ │ 192.168.1.13 │
└───────────────────┘ └───────────────────┘ └───────────────────┘ n1 & n2 & n3 --- lan(("LAN<br/>192.168.1.0/24"))
│ │ │
────┴──────────────────────┴──────────────────────┴──── classDef mgmt fill:#CC79A7,color:#fff,stroke:#a36088
LAN (192.168.1.0/24) classDef instance fill:#56B4E9,color:#fff,stroke:#3a8fbf
classDef network fill:#0072B2,color:#fff,stroke:#005a8e
classDef node fill:#009E73,color:#fff,stroke:#007a5e
class ovnc mgmt
class ctrl1,ctrl2,ctrl3 node
class c1,c2,c3,c4 instance
class ls1,ls2,ls3,lan network
style cp fill:#f5e6f0,stroke:#CC79A7
style n1 fill:#e6f5f0,stroke:#009E73
style n2 fill:#e6f5f0,stroke:#009E73
style n3 fill:#e6f5f0,stroke:#009E73
``` ```
**Key difference from bridge:** All instances share a single logical switch **Key difference from bridge:** All instances share a single logical switch

91
notes/observability-guide.md
View File

@ -11,57 +11,48 @@ traffic on the shared network.
## Architecture ## Architecture
``` ```mermaid
LAN (192.168.103.0/24) flowchart TD
| lan(["LAN 192.168.103.0/24"])
| OVN network forward (192.168.103.201)
| :3000 -> 10.10.10.70:3000 (Grafana)
| :9090 -> 10.10.10.70:9090 (Prometheus)
|
═══════╪══════════════════════════════════════════════
| OVN net-prod (10.10.10.0/24)
|
v
+----------------------------------------------+
| monitoring (10.10.10.70, oc-node-02) |
| Debian/12 container — 2 GiB RAM, 20 GiB |
| |
| +-------------+ +-----------+ |
| | Prometheus | | Grafana | |
| | :9090 | | :3000 | |
| +-------------+ +-----------+ |
| +-------------+ +-----------+ |
| | Loki | | Promtail | |
| | :3100 | | (agent) | |
| | gRPC :9096 | | | |
| +-------------+ +-----------+ |
+----------------------------------------------+
|
| Scrape targets
|
+-------> Incus nodes (client cert auth, :8443)
| 192.168.102.140 /1.0/metrics
| 192.168.102.141 /1.0/metrics
| 192.168.102.142 /1.0/metrics
|
+-------> HAProxy (:8404)
| 10.10.10.50 /metrics
| 10.10.10.51 /metrics
|
+-------> node-exporters (:9100)
10.10.10.71 node-exp-01 (oc-node-01)
10.10.10.72 node-exp-02 (oc-node-02)
10.10.10.73 node-exp-03 (oc-node-03)
+------------------+ +------------------+ +------------------+ subgraph fwd["OVN Forward · 192.168.103.201"]
| node-exp-01 | | node-exp-02 | | node-exp-03 | fwd_grafana[":3000 → Grafana"]
| 10.10.10.71 | | 10.10.10.72 | | 10.10.10.73 | fwd_prom[":9090 → Prometheus"]
| Alpine, privil. | | Alpine, privil. | | Alpine, privil. | end
| :9100 | | :9100 | | :9100 |
| host /proc,/sys | | host /proc,/sys | | host /proc,/sys | subgraph mon["monitoring · 10.10.10.70 · oc-node-02"]
| 128 MiB RAM | | 128 MiB RAM | | 128 MiB RAM | prometheus["Prometheus :9090"]
| pinned to node | | pinned to node | | pinned to node | grafana["Grafana :3000"]
+------------------+ +------------------+ +------------------+ loki["Loki :3100<br/>gRPC :9096"]
promtail["Promtail"]
end
subgraph targets["Scrape Targets"]
incus["Incus nodes :8443<br/>.140 · .141 · .142"]
haproxy["HAProxy :8404<br/>.50 · .51"]
ne["node-exporters :9100<br/>.71 · .72 · .73"]
end
lan --> fwd
fwd --> mon
prometheus -->|"client cert"| incus
prometheus --> haproxy
prometheus --> ne
classDef external fill:#f5f5f5,color:#333,stroke:#999
classDef network fill:#0072B2,color:#fff,stroke:#005a8e
classDef mgmt fill:#CC79A7,color:#fff,stroke:#a36088
classDef instance fill:#56B4E9,color:#fff,stroke:#3a8fbf
classDef node fill:#009E73,color:#fff,stroke:#007a5e
class lan external
class fwd_grafana,fwd_prom network
class prometheus,grafana,loki,promtail mgmt
class incus,haproxy,ne instance
style fwd fill:#e0eef8,stroke:#0072B2
style mon fill:#f5e6f0,stroke:#CC79A7
style targets fill:#e0f2fe,stroke:#56B4E9
``` ```
## Components ## Components

60
notes/operations-center-guide.md
View File

@ -9,39 +9,33 @@ build 202602230420 on Proxmox VE 9.1.5 (nested virtualization on Intel).
## Section 0: Architecture Overview ## Section 0: Architecture Overview
``` ```mermaid
┌──────────────────────────────────────────┐ flowchart TD
│ Proxmox VE Host (pve) │ subgraph proxmox["Proxmox VE Host · i9-13900HK · 64 GiB"]
│ Intel i9-13900HK 64 GiB 20 cores │ subgraph cluster["Incus Cluster · net-prod 10.10.10.0/24"]
│ │ n1["oc-node-01<br/>VMID 400 · .140"]
│ ┌──────────┐ ┌──────────┐ ┌──────────┐│ n2["oc-node-02<br/>VMID 401 · .141"]
│ │oc-node-01│ │oc-node-02│ │oc-node-03││ n3["oc-node-03<br/>VMID 402 · .142"]
│ │ VMID 400 │ │ VMID 401 │ │ VMID 402 ││ end
│ │ 4c/8G/64G│ │ 4c/8G/50G│ │ 4c/8G/50G││ oc["Operations Center<br/>VMID 920 · .120"]
│ │ .140 │ │ .141 │ │ .142 ││ end
│ │ │ │ │ │ ││
│ │ovn-central│ │ │ │ ││ vmbr0(("VLAN 69<br/>192.168.100.0/22"))
│ │ha-web-01 │ │ha-web-02 │ │ha-web-03 ││ ext["OVN external IPs<br/>192.168.103.200-210"]
│ └──────────┘ └──────────┘ └──────────┘│
│ │ │ │ │ proxmox --- vmbr0
│ └────────────┼────────────┘ │ cluster -.->|"external gateway"| ext
│ OVN Geneve tunnels │
│ net-prod 10.10.10.0/24 │ classDef node fill:#009E73,color:#fff,stroke:#007a5e
│ │ classDef mgmt fill:#CC79A7,color:#fff,stroke:#a36088
│ ┌──────────┐ │ classDef network fill:#0072B2,color:#fff,stroke:#005a8e
│ │oc-server │ OC Management │
│ │ VMID 920 │ https://192.168.102.120 │ class n1,n2,n3 node
│ │ 2c/4G/50G│ Web UI + CLI + REST API │ class oc mgmt
│ │ .120 │ │ class vmbr0,ext network
│ └──────────┘ │
│ │ style proxmox fill:#f5f5f5,stroke:#999
│ vmbr0 (VLAN 69) ── 192.168.100.0/22 │ style cluster fill:#e6f5f0,stroke:#009E73
└──────────────────────────────────────────┘
│
│ gateway 192.168.100.1
│
OVN external IPs: 192.168.103.200-210
LB: 192.168.103.201 Fwd: 192.168.103.202
``` ```
### Infrastructure ### Infrastructure

78
notes/ovn-deep-dive.md
View File

@ -6,16 +6,25 @@ cluster running OVS 3.6.1 with an OVN overlay network.
## Lab Topology ## Lab Topology
``` ```mermaid
LAN (192.168.100.0/22) flowchart TD
│ lan(("LAN<br/>192.168.100.0/22"))
├── oc-node-01 192.168.102.140 (Geneve endpoint)
├── oc-node-02 192.168.102.141 (Geneve endpoint) n1["oc-node-01 · .140<br/>Geneve endpoint"]
├── oc-node-03 192.168.102.142 (Geneve endpoint, gateway chassis) n2["oc-node-02 · .141<br/>Geneve endpoint"]
│ n3["oc-node-03 · .142<br/>Geneve + gateway"]
└── UPLINK physical network
└── OVN range: 192.168.103.200-210 uplink("UPLINK<br/>OVN range .103.200-210")
└── net-prod (10.10.10.0/24) ← OVN overlay netprod("net-prod<br/>10.10.10.0/24")
lan --- n1 & n2 & n3
lan --- uplink --> netprod
classDef node fill:#009E73,color:#fff,stroke:#007a5e
classDef network fill:#0072B2,color:#fff,stroke:#005a8e
class n1,n2,n3 node
class lan,uplink,netprod network
``` ```
**Instances on net-prod:** **Instances on net-prod:**
@ -302,14 +311,30 @@ Each IncusOS node runs OVS 3.6.1 with two bridges.
### Bridge Architecture ### Bridge Architecture
``` ```mermaid
Per-node OVS layout: flowchart LR
subgraph provider["incusovn7 · provider bridge"]
nic["physical NIC"]
intport["internal port"]
patch1["patch to br-int"]
end
incusovn7 (provider bridge) br-int (integration bridge) subgraph integration["br-int · integration bridge"]
├── incusovn7 ← physical NIC ├── veth* ← instance NICs veth["instance veth ports"]
├── incusovn7b ← internal port ├── ovn-* ← Geneve tunnels ovntun["Geneve tunnels"]
└── patch-...-to-br-int ←──patch──→ ├── patch-br-int-to-... patch2["patch to incusovn7"]
└── br-int ← internal port end
patch1 <-->|"patch port"| patch2
classDef prov fill:#0072B2,color:#fff,stroke:#005a8e
classDef integ fill:#009E73,color:#fff,stroke:#007a5e
class nic,intport,patch1 prov
class veth,ovntun,patch2 integ
style provider fill:#e0eef8,stroke:#0072B2
style integration fill:#e6f5f0,stroke:#009E73
``` ```
**`br-int` (integration bridge)**: **`br-int` (integration bridge)**:
@ -329,12 +354,19 @@ incusovn7 (provider bridge) br-int (integration bridge)
Every pair of chassis has a Geneve tunnel with BFD health monitoring: Every pair of chassis has a Geneve tunnel with BFD health monitoring:
``` ```mermaid
oc-node-01 ←── Geneve (UDP 6081) ──→ oc-node-02 graph LR
↑ ↑ n1(("oc-node-01<br/>.140"))
└──────── Geneve (UDP 6081) ──────────┘ n2(("oc-node-02<br/>.141"))
↓ n3(("oc-node-03<br/>.142"))
oc-node-03
n1 <-->|"Geneve 6081"| n2
n2 <-->|"Geneve 6081"| n3
n1 <-->|"Geneve 6081"| n3
classDef chassis fill:#009E73,color:#fff,stroke:#007a5e
class n1,n2,n3 chassis
``` ```
| Source | Destination | OVS Port Name | BFD State | | Source | Destination | OVS Port Name | BFD State |

63
notes/production-lab-guide.md
View File

@ -14,35 +14,40 @@ client 6.21, Operations Center v0.3.0.
### Network Topology ### Network Topology
``` ```mermaid
VLAN 69 (192.168.100.0/22) flowchart TD
Gateway: 192.168.100.1 vlan(("VLAN 69<br/>192.168.100.0/22"))
│
┌──────────────────┼──────────────────────────────────┐ subgraph mgmt["Management"]
│ │ │ oc["lab-oc<br/>VMID 910 · .110<br/>OC server"]
┌──────┴──────┐ ┌──────┴──────┐ ┌───────┴─────┐ ┌─────┴───────┐ end
│ lab-oc │ │ lab-node-01 │ │ lab-node-02 │ │ lab-node-03 │
│ VMID 910 │ │ VMID 911 │ │ VMID 912 │ │ VMID 913 │ subgraph cluster["Incus Cluster"]
│ .102.110 │ │ .102.111 │ │ .102.112 │ │ .102.113 │ n1["lab-node-01<br/>VMID 911 · .111<br/>init + ovn-central"]
│ OC server │ │ Init node │ │ Join node │ │ Join node │ n2["lab-node-02<br/>VMID 912 · .112"]
│ 2c/4G/50G │ │ 4c/8G/64G │ │ 4c/8G/50G │ │ 4c/8G/50G │ n3["lab-node-03<br/>VMID 913 · .113"]
└─────────────┘ │ │ │ │ │ │ end
│ ovn-central │ │ │ │ │
│ (container) │ │ │ │ │ subgraph networks["OVN Networks"]
│ NB:6641 │ │ │ │ │ prod("net-prod<br/>10.10.10.0/24")
│ SB:6642 │ │ │ │ │ iso("net-isolated<br/>10.10.20.0/24")
└─────────────┘ └─────────────┘ └─────────────┘ uplink("UPLINK<br/>.103.200-210")
│ │ │ end
└────────────────┼────────────────┘
Geneve tunnels (OVN) vlan --- mgmt & cluster
│ cluster -->|"Geneve tunnels"| networks
┌────────────────┼────────────────┐
│ │ │ classDef nodeClass fill:#009E73,color:#fff,stroke:#007a5e
┌──────┴──────┐ ┌──────┴──────┐ ┌─────┴───────┐ classDef mgmtClass fill:#CC79A7,color:#fff,stroke:#a36088
│ net-prod │ │net-isolated │ │ UPLINK │ classDef networkClass fill:#0072B2,color:#fff,stroke:#005a8e
│ 10.10.10/24 │ │ 10.10.20/24 │ │ .103.200-210│
│ OVN L2 │ │ OVN L2 │ │ Physical │ class n1,n2,n3 nodeClass
└─────────────┘ └─────────────┘ └─────────────┘ class oc mgmtClass
class prod,iso,uplink,vlan networkClass
style mgmt fill:#f5e6f0,stroke:#CC79A7
style cluster fill:#e6f5f0,stroke:#009E73
style networks fill:#e0eef8,stroke:#0072B2
``` ```
### Infrastructure ### Infrastructure

128
notes/shared-storage-guide.md
View File

@ -43,21 +43,31 @@ on local ZFS.
### How iSCSI + lvmcluster works ### How iSCSI + lvmcluster works
``` ```mermaid
┌──────────────────┐ iSCSI (1GbE) ┌──────────────────┐ flowchart LR
│ iSCSI Target │◄───────────────────────►│ oc-node-01 │ target["iSCSI Target<br/>QNAP / VM / any target<br/>Exposes LUN as block device"]
│ (QNAP / VM / │◄───────────────────────►│ oc-node-02 │
│ any target) │◄───────────────────────►│ oc-node-03 │ subgraph cluster["Incus Cluster Nodes"]
│ │ │ │ n1["oc-node-01"]
│ Exposes LUN(s) │ │ iSCSI initiator │ n2["oc-node-02"]
│ as block device │ │ (IncusOS built-in)│ n3["oc-node-03"]
└──────────────────┘ │ │ stack["iSCSI initiator (built-in)<br/>lvmlockd + sanlock (built-in)<br/>Incus lvmcluster pool"]
│ lvmlockd+sanlock│ end
│ (IncusOS built-in)│
│ │ target <-->|"iSCSI (1GbE)"| n1
│ Incus lvmcluster│ target <-->|"iSCSI (1GbE)"| n2
│ storage pool │ target <-->|"iSCSI (1GbE)"| n3
└──────────────────┘ n1 & n2 & n3 ~~~ stack
classDef network fill:#0072B2,color:#fff,stroke:#005a8e
classDef node fill:#009E73,color:#fff,stroke:#007a5e
classDef instance fill:#56B4E9,color:#fff,stroke:#3a8fbf
class target network
class n1,n2,n3 node
class stack instance
style cluster fill:#e6f5f0,stroke:#009E73
``` ```
1. **iSCSI target** exports a block device (LUN) over the network 1. **iSCSI target** exports a block device (LUN) over the network
@ -97,29 +107,33 @@ The recommended setup uses **both** pools:
### Network topology (tested) ### Network topology (tested)
``` ```mermaid
┌─ Proxmox host ──────────────────────────────────────────────────────┐ flowchart TD
│ │ subgraph proxmox["Proxmox Host"]
│ ┌──────────────────────────────────────────────────────────────┐ │ subgraph n1["oc-node-01 · VMID 400 · .140"]
│ │ oc-node-01 (VMID 400, 192.168.102.140) │ │ target["iscsi-target<br/>10.207.217.19 (bridge)<br/>192.168.102.150 (macvlan)<br/>LUN: 20 GiB"]
│ │ ┌──────────────┐ │ │ end
│ │ │iscsi-target │ Debian container running tgt │ │
│ │ │10.207.217.19 │ incusbr0 (bridge, same-node access) │ │ n2["oc-node-02<br/>VMID 401 · .141<br/>lvmcluster"]
│ │ │.102.150 │ macvlan on mgmt (cross-node access) │ │ n3["oc-node-03<br/>VMID 402 · .142<br/>lvmcluster"]
│ │ │LUN: 20 GiB │◄─── iSCSI initiator (oc-node-01 via bridge)│ │ end
│ │ └──────────────┘◄─── iSCSI initiator (oc-node-02 via macvlan)│ │
│ │ ◄─── iSCSI initiator (oc-node-03 via macvlan)│ │ target <-->|"bridge"| n1
│ └──────────────────────────────────────────────────────────────┘ │ target <-->|"macvlan"| n2
│ │ target <-->|"macvlan"| n3
│ ┌──────────┐ ┌──────────┐ │ vlan(("VLAN 69<br/>192.168.100.0/22"))
│ │oc-node-02│ │oc-node-03│ │ proxmox --- vlan
│ │ VMID 401 │ │ VMID 402 │ │
│ │ .102.141 │ │ .102.142 │ │ classDef network fill:#0072B2,color:#fff,stroke:#005a8e
│ │ lvmclust │ │ lvmclust │ │ classDef node fill:#009E73,color:#fff,stroke:#007a5e
│ └──────────┘ └──────────┘ │ classDef lb fill:#E69F00,color:#fff,stroke:#b87d00
│ │
│ vmbr0 (VLAN 69) ── 192.168.100.0/22 │ class target network
└─────────────────────────────────────────────────────────────────────┘ class n2,n3 node
class vlan network
style proxmox fill:#f5f5f5,stroke:#999
style n1 fill:#e6f5f0,stroke:#009E73
``` ```
**Lab target container** (Option A, tested): a Debian container on oc-node-01 **Lab target container** (Option A, tested): a Debian container on oc-node-01
@ -1365,19 +1379,31 @@ For large VMs, this can take time depending on disk size and pool speeds.
With shared iSCSI storage, adding a second Proxmox host enables true cross- With shared iSCSI storage, adding a second Proxmox host enables true cross-
host live migration: host live migration:
``` ```mermaid
┌─ Proxmox Host A ────────┐ ┌─ Proxmox Host B ────────┐ flowchart TD
│ oc-node-01, oc-node-02 │ │ oc-node-03, oc-node-04 │ subgraph hostA["Proxmox Host A"]
│ iSCSI initiators │ │ iSCSI initiators │ a1["oc-node-01"]
└──────────┬───────────────┘ └──────────┬───────────────┘ a2["oc-node-02"]
│ │ end
└────────── iSCSI ───────────────┘
│ subgraph hostB["Proxmox Host B"]
┌─────────┴──────────┐ b1["oc-node-03"]
│ QNAP NAS │ b2["oc-node-04"]
│ iSCSI target │ end
│ Shared LUN │
└────────────────────┘ qnap["QNAP NAS<br/>iSCSI target<br/>Shared LUN"]
hostA <-->|"iSCSI"| qnap
hostB <-->|"iSCSI"| qnap
classDef node fill:#009E73,color:#fff,stroke:#007a5e
classDef network fill:#0072B2,color:#fff,stroke:#005a8e
class a1,a2,b1,b2 node
class qnap network
style hostA fill:#f5f5f5,stroke:#999
style hostB fill:#f5f5f5,stroke:#999
``` ```
Requirements: Requirements: