maarten
/
incus-contrib
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
incus-contrib/notes/aether-help-texts.md

15552 lines
431 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Aether Help Texts
All help content extracted from Aether (FFSDN) web application at
`https://192.168.102.160:8443`. These are built-in documentation pages
accessible via Help buttons throughout the UI.
**Extracted**: 2026-02-23
**Pages found**: 16
## Table of Contents
1. [Incus – Clusters](#incus--clusters)
2. [Incus – Cluster Groups](#incus--cluster-groups)
3. [Incus – Storage Pools](#incus--storage-pools)
4. [Incus – Storage Volumes](#incus--storage-volumes)
5. [Incus – Custom Isos](#incus--custom-isos)
6. [Incus – Storage Buckets](#incus--storage-buckets)
7. [Incus – Profiles](#incus--profiles)
8. [Incus – Settings](#incus--settings)
9. [Incus – Images](#incus--images)
10. [Incus – Configuration](#incus--configuration)
11. [Incus – Instances](#incus--instances)
12. [Deployhelp](#deployhelp)
13. [Healthhelp](#healthhelp)
14. [Traceflowhelp](#traceflowhelp)
15. [NewACL](#newacl)
16. [OVNHAproxyLB](#ovnhaproxylb)
---
## Incus – Clusters
*Source: `/static/help_incus_clusters.html`*
AETHER INCUS Cluster Members Guide
## AETHER INCUS Cluster Members Guide
This guide explains how Incus clustering works, what the Cluster Members tab shows, and how to use the Evacuate and Restore operations for maintenance.
**Table of Contents**
- What Is an Incus Cluster?
- Cluster Architecture
- Member Roles and the Distributed Database
- Quorum: How the Cluster Stays Alive
- Members Table Columns
- Member Statuses
- Member Detail View
- Evacuating a Cluster Member
- Evacuation Modes Explained
- Restoring a Cluster Member
- Automatic Healing
- Best Practices
### What Is an Incus Cluster?
An Incus cluster is a group of servers (called **members**) that work together as a single system. Instead of managing each server individually, you manage one cluster — and Incus takes care of distributing instances (containers and virtual machines) across the available members.
Key benefits of clustering:
- **High availability** — If one member goes down, the cluster continues to operate. Instances on shared storage can be moved to other members.
- **Workload distribution** — Instances are spread across members to balance resource usage. New instances are automatically placed on the member with the fewest running instances.
- **Unified management** — All members share the same configuration, images, profiles, and networks. You manage everything from a single API endpoint.
- **Live maintenance** — Members can be evacuated for maintenance without downtime for the overall cluster.
### Cluster Architecture
An Incus cluster consists of one **bootstrap server** (the first member that initialises the cluster) and at least two additional members. While two-member clusters are technically possible, **three or more members are strongly recommended** for fault tolerance.
All cluster members share a **distributed database** (powered by Cowsql, which uses the Raft consensus algorithm). This database stores all cluster state: instance definitions, network configurations, storage pools, certificates, and more.
Members communicate with each other via **heartbeats** — periodic health checks sent between members. If a member fails to send heartbeats for a configurable period (default: 20 seconds), it is marked as offline.
Cluster Overview
+-----------------+ +-----------------+ +-----------------+
| Member 1 | | Member 2 | | Member 3 |
| (voter) |<--->| (voter) |<--->| (voter) |
| DB Leader | | DB Replica | | DB Replica |
| | | | | |
| instance-a | | instance-c | | instance-e |
| instance-b | | instance-d | | instance-f |
+-----------------+ +-----------------+ +-----------------+
^ ^ ^
| Heartbeats | |
+-----------------------+-----------------------+
+-----------------+ +-----------------+
| Member 4 | | Member 5 |
| (stand-by) | | (spare) |
| DB Replica | | No DB role |
| | | |
| instance-g | | instance-h |
+-----------------+ +-----------------+
### Member Roles and the Distributed Database
Not every cluster member participates in the distributed database equally. Members are assigned roles that determine their level of involvement:
|
Role |
DB Participation |
Description |
|
**database-leader** |
Full (leader) |
The current leader of the distributed database. All writes go through this member first. There is always exactly **one** leader. If the leader goes offline, a new leader is elected from the voters. |
|
**database** |
Full (voter) |
A **voter** — participates in database consensus. Voters must acknowledge writes before they are committed. The number of voters is controlled by `cluster.max_voters` (default: 3). |
|
**database-standby** |
Replica (non-voting) |
A **stand-by** member that maintains a copy of the database but does not vote. If a voter goes offline, a stand-by is automatically **promoted to voter**. The number of stand-by members is controlled by `cluster.max_standby` (default: 2). |
|
*(no database role)* |
None |
A **spare** member. It runs instances but does not participate in the database at all. In very large clusters, most members are spares — they contribute compute and storage resources without adding database overhead. |
|
**event-hub** |
N/A |
An optional role that centralises event distribution. In large clusters, event-hub members reduce the number of connections needed for event propagation. |
|
**ovn-chassis** |
N/A |
An optional role indicating this member participates in OVN network chassis traffic. This is relevant for OVN gateway functionality. |
**Tip:** Click on any member's **Server Name** in the table to open a detail view showing the full hardware specifications, including CPU details, memory breakdown, network interfaces, storage disks, and GPU information.
### Quorum: How the Cluster Stays Alive
The cluster's distributed database uses the **Raft consensus algorithm**, which requires a **majority of voters** (called a *quorum*) to be online for the database to accept writes. This is the most critical concept to understand about clustering:
|
Number of Voters |
Quorum Required |
Can Tolerate Failures |
|
3 |
2 |
1 voter offline |
|
5 |
3 |
2 voters offline |
|
7 |
4 |
3 voters offline |
With the default of 3 voters and 2 stand-by members:
- If **1 voter** goes offline → a stand-by is promoted to voter, quorum is maintained, cluster operates normally.
- If **2 voters** go offline simultaneously → quorum is lost, the database becomes read-only, no new instances can be created or modified.
**Critical:** If quorum is lost (majority of voters are offline), the cluster's database becomes read-only. Running instances continue to run, but no management operations are possible until enough voters come back online to restore quorum. This is why having stand-by members is essential — they provide automatic promotion to restore quorum quickly.
### Members Table Columns
The Cluster Members table displays the following information for each member. All columns are sortable by clicking on the column header.
|
Column |
Description |
|
**Server Name** |
The hostname of the cluster member. This is a clickable link that opens the **Member Detail** view with full hardware information. The name is assigned when the member joins the cluster and cannot be changed afterwards. |
|
**URL** |
The API URL used to reach this member (e.g., `https://192.168.1.10:8443`). Other members use this URL for internal communication (heartbeats, database replication, image sync). This corresponds to the `cluster.https_address` setting. |
|
**Status** |
The current operational status of the member. See Member Statuses below for all possible values and their colour coding. |
|
**Memory** |
A visual usage bar showing RAM consumption on this member.
Format: `XX% (used GB / total GB)`
Colour thresholds:
- **Green** — Below 70% usage
- **Orange** — 70% to 89% usage
- **Red** — 90% and above
Memory values reflect real-time usage on the physical host, including all running instances, the OS, and any other processes.
|
|
**Load** |
A visual usage bar showing CPU load as a percentage of total CPU threads.
Format: `XX% (1-min load average / total threads)`
The value is calculated as: `(load_avg_1min / cpu_threads) × 100`
Colour thresholds (same as Memory):
- **Green** — Below 70%
- **Orange** — 70% to 89%
- **Red** — 90% and above
|
|
**Message** |
A status message from Incus about this member. Common values:
- `Fully operational` — Everything is working normally.
- `No heartbeat since ...` — The member has stopped responding. The cluster is waiting for the offline threshold before marking it offline.
- `Evacuated` — The member has been evacuated and is in maintenance mode.
|
|
**Roles** |
The database and cluster roles assigned to this member (e.g., `database-leader`, `database`, `database-standby`, `event-hub`, `ovn-chassis`). See Member Roles above for detailed descriptions. Members without any database role are spare members. |
|
**Architecture** |
The CPU architecture of this member (e.g., `x86_64`, `aarch64`). Incus supports mixed-architecture clusters, but instances can only run on members with a matching architecture. |
|
**Actions** |
Available operations for this member:
- **Evacuate** (orange button) — Shown when the member is Online. Starts the evacuation process.
- **Restore** (green button) — Shown when the member is Evacuated. Returns the member to normal operation.
- **In Progress...** (disabled) — Shown while an evacuation or restore operation is running.
- **-** — Shown for offline members (no action available) or for auditor users (read-only access).
|
#### Understanding the Load Column
The Load column shows the **1-minute load average** as a percentage of available CPU threads. The load average is a Unix metric that represents the average number of processes waiting for or actively using the CPU over the last 1 minute.
- A load of **100%** means the number of busy processes equals the number of CPU threads — the system is fully utilised.
- A load **above 100%** means there are more processes wanting CPU time than there are threads available — processes are queueing. The bar caps at 100% width but the percentage label will show the actual value.
- A load of **50%** on a 16-thread system means an average of 8 threads are busy — the system has headroom.
**Tip:** Brief load spikes above 100% are normal during batch operations. Sustained high load indicates the member may need more CPU resources or some instances should be moved to other members.
### Member Statuses
Each cluster member has a status that reflects its current operational state:
|
Status |
Colour |
Meaning |
|
Online |
Green |
The member is healthy, responding to heartbeats, and accepting workloads. This is the normal operating state. |
|
Evacuated |
Yellow |
The member has been intentionally evacuated. All its instances have been moved to other members. No new instances will be placed on this member. The member is typically undergoing maintenance (e.g., OS updates, hardware replacement). |
|
Evacuating... |
Yellow (pulsing) |
An evacuation operation is currently in progress. Instances are being moved off this member. The pulsing animation indicates an active long-running operation. |
|
Restoring... |
Yellow (pulsing) |
A restore operation is currently in progress. Instances are being moved back to this member. |
|
Offline |
Red |
The member has not sent a heartbeat within the `cluster.offline_threshold` period (default: 20 seconds). The member is unreachable — it may have crashed, lost network connectivity, or been shut down unexpectedly. No operations can be performed on this member until it comes back online. |
|
Blocked |
Grey |
The member is blocked from accepting API requests. This typically occurs during a cluster upgrade when this member's Incus version does not match the other members. Running instances continue to operate, but no management operations are possible. |
### Member Detail View
Clicking a **Server Name** in the members table opens a detailed modal with two tabs:
#### Overview Tab
Shows key information about the member:
- **Status**, **Message**, **Roles**, **Architecture**, **URL**
- **OS** and **Kernel version**
- **Uptime** — How long the server has been running since last boot
- **CPU** — Total threads, usage bar, and core count
- **Memory** — Usage bar with used/total values
- **Swap** — Swap usage if configured
#### Hardware Tab
Shows detailed hardware information retrieved from the member:
- **CPU** — Processor model, socket count, core count, thread count, architecture, and detailed per-socket information
- **Memory** — Total RAM, individual DIMM details (size, type, speed, slot)
- **Network Interfaces** — Each NIC with its driver, MAC address, link speed, and associated ports
- **Storage Disks** — Each physical disk with model, serial number, size, type (HDD/SSD/NVMe), RPM, and partitions
- **GPUs** — If present, GPU vendor, model, driver, NUMA node, and associated cards/devices
### Evacuating a Cluster Member
Evacuation is the process of **moving all instances off a cluster member** so you can safely perform maintenance on it. When you click the **Evacuate** button on a member, a modal opens where you choose the evacuation mode.
#### What Happens During Evacuation
- Incus processes each instance on the member one by one.
- Depending on the chosen evacuation mode, each instance is either migrated (moved live or after stopping) to another member, or stopped in place.
- Instances are distributed across remaining members based on available resources.
- The member's status changes to **Evacuated**.
- The member is blocked from receiving any new instances.
The operation runs asynchronously. A progress indicator shows the elapsed time, and the status badge pulses while the operation is in progress. Depending on the number and size of instances, evacuation can take anywhere from seconds to many minutes.
**Note:** Instances are shut down cleanly during evacuation, respecting the `boot.host_shutdown_timeout` instance configuration (how long to wait for a graceful shutdown before forcing). The default is 5 minutes.
### Evacuation Modes Explained
The evacuation mode determines *how* instances are moved off the member. Different modes offer different trade-offs between downtime and safety:
|
Mode |
Description |
Downtime |
Best For |
|
**Auto**
*(recommended)* |
Incus decides the best action for each instance individually. Virtual machines with `migration.stateful=true` are live-migrated. Other instances are stopped then migrated. Instances with local devices that prevent migration are stopped in place. |
Varies per instance |
Most situations. Safe default that handles mixed workloads intelligently. |
|
**Live Migrate** |
Instances are moved to another member **while still running**. For virtual machines, this means near-zero downtime — the VM's memory is copied to the target while it continues to run, with only a brief pause at the final switchover (typically milliseconds to a few seconds). For containers, live migration support is limited. |
Near-zero (VMs) |
Production VMs where downtime is unacceptable. Requires `migration.stateful=true` on the instance. |
|
**Migrate** |
Instances are **stopped first**, then their disk and configuration are migrated to another member, and the instance is started there. This is a cold migration. |
Full stop/start cycle |
When live migration is not possible or not needed. Safest option for data integrity. |
|
**Stop** |
Instances are **gracefully shut down** (like pressing the power button). The shutdown respects `boot.host_shutdown_timeout`. Instances are then migrated to another member. |
Full stop/start cycle |
When you want instances to go through their normal shutdown procedure (e.g., to flush caches, close database connections). |
|
**Stateful Stop** |
The instance's **runtime state (RAM contents) is saved to disk** before stopping. When restored, the instance resumes exactly where it left off — similar to hibernation on a laptop. This is supported for VMs; container support is limited. |
Pause + resume |
When you need to preserve running application state (e.g., in-memory data that would be lost on restart). |
|
**Force Stop** |
Instances are **forcefully killed** immediately (equivalent to pulling the power cord). No graceful shutdown is attempted. Instances are then migrated. |
Full stop/start cycle |
Emergency situations where instances are unresponsive and a graceful shutdown is not possible. Use with caution — may cause data loss. |
**Warning:** **Live Migrate** requires `migration.stateful=true` to be set on the instance configuration *before* attempting the evacuation. If this setting is not enabled, the live migration will fail or fall back to a cold migration. Live migration also requires that all of the instance's devices (disks, network interfaces) are compatible with migration.
**Caution:** **Force Stop** does not give instances any time to shut down gracefully. This can result in data corruption for applications that have data in write caches or open database transactions. Only use this mode when absolutely necessary.
### Restoring a Cluster Member
Once maintenance is complete and the member is back online, click the **Restore** button (green) to return it to normal operation.
#### What Happens During Restore
- Incus identifies all instances that were originally on this member before evacuation.
- Those instances are **migrated back** from their temporary locations to this member.
- The member's status changes from **Evacuated** back to **Online**.
- The member starts accepting new instance placements again.
Like evacuation, the restore operation runs asynchronously and displays a progress indicator with elapsed time.
**Note:** Restore moves instances back to the member they came from. This ensures your workload distribution returns to its pre-maintenance state. The instances that were moved during evacuation remember their original member and are returned there.
### Automatic Healing
Incus supports **automatic evacuation** of offline members. When a member goes offline (e.g., hardware failure) and remains offline for longer than the `cluster.healing_threshold` setting, Incus will automatically evacuate its instances to other members.
By default, automatic healing is **disabled** (`cluster.healing_threshold = 0`). To enable it, set the threshold to the number of seconds after which automatic evacuation should trigger.
**Important considerations for automatic healing:**
- Automatic healing only works for instances on **shared storage** (e.g., Ceph, LINSTOR). Instances on local storage cannot be automatically moved because the disk data is not accessible from other members.
- Set the threshold high enough to avoid triggering during **planned reboots** (e.g., 300 seconds or more). A member rebooting for a kernel update would otherwise have all its instances evacuated unnecessarily.
- If the member comes back online after automatic healing, its instances will **not** automatically return. You must manually run a Restore to bring them back.
- Be cautious with automatic healing during **network partitions** (split-brain scenarios). If a member is actually running but unreachable from the rest of the cluster, evacuating its instances could lead to conflicts.
### Best Practices
#### Planned Maintenance Workflow
- **Verify** that all other cluster members are Online and healthy.
- **Evacuate** the member using Auto mode (recommended) or the appropriate mode for your workload.
- **Wait** for the evacuation to complete. Monitor the progress in AETHER.
- **Perform maintenance** on the member (OS updates, hardware changes, etc.).
- **Verify** the member is back online (status shows Online with the message "Fully operational").
- **Restore** the member to bring instances back.
#### Cluster Sizing Recommendations
|
Cluster Size |
Recommended Voters |
Recommended Stand-by |
Tolerates |
|
3 members |
3 |
0 |
1 member offline |
|
4-5 members |
3 |
1-2 |
1 voter offline (stand-by auto-promotes) |
|
6-10 members |
3 |
2 |
1 voter + 1 stand-by offline |
|
10+ members |
5 |
2 |
2 voters offline simultaneously |
#### Monitoring Tips
- **Watch the Memory column** — Members consistently above 70% may need capacity added or instances rebalanced. Above 90% is a warning sign that the member may start swapping, which severely degrades performance.
- **Watch the Load column** — Sustained load above 70% indicates CPU contention. Instances may be experiencing slower response times.
- **Check Roles regularly** — Ensure you always have the expected number of voters and stand-by members. If a voter was lost and a stand-by was promoted, the cluster may now have fewer stand-by members than expected.
- **Never evacuate more than one member at a time** in small clusters. Ensure the first evacuation is complete before starting another.
- **Monitor Message column** — "No heartbeat since..." messages indicate a member is struggling. Investigate before it goes fully offline.
**Related settings:** The cluster behaviour can be tuned via the Settings tab. Key settings include `cluster.offline_threshold` (how quickly offline is detected), `cluster.healing_threshold` (automatic evacuation), `cluster.max_voters` and `cluster.max_standby` (database redundancy). See the Settings tab Help for full details.
---
## Incus – Cluster Groups
*Source: `/static/help_incus_cluster_groups.html`*
AETHER INCUS Cluster Groups Guide
## AETHER INCUS Cluster Groups Guide
This guide explains what Incus cluster groups are, how they work, and how to manage them in AETHER Infrastructure Management.
#### Table of Contents
- 1. What Are Cluster Groups?
- 2. Why Use Cluster Groups?
- 3. The Default Cluster Group
- 4. Instance Placement & Scheduling
-
- 4.1 Targeting a Cluster Group
- 4.2 Scheduler Modes
- 4.3 How Incus Selects a Member
- 5. Evacuation & Cluster Groups
- 6. Cluster Group Configuration
- 7. Common Use Cases
- 8. Table Columns Explained
- 9. Creating a Cluster Group
- 10. Editing a Cluster Group
- 11. Deleting a Cluster Group
- 12. Best Practices
### 1. What Are Cluster Groups?
In an Incus cluster, every server that participates is called a **cluster member**. A **cluster group** is a way to organise one or more of these members into a named, logical subset. Instead of treating every server in the cluster as interchangeable, cluster groups let you label servers that share a common characteristic and then direct workloads to those specific servers.
Think of cluster groups as tags or labels for your servers. A group called `gpu` might contain the three servers that have GPUs installed. A group called `production` might contain the servers designated for production workloads. When you launch a new instance, you can say "put this on a server in the `gpu` group" rather than picking a specific server by name.
Key characteristics of cluster groups:
- A cluster group has a **name**, an optional **description**, and a list of **members**.
- A single cluster member can belong to **multiple groups** simultaneously.
- Every cluster automatically has a **default** group that all members join when they are added to the cluster.
- Cluster groups can optionally carry **configuration keys** (such as CPU baseline settings for VM live migration).
Incus Cluster
+-----------------------------------------------+
| |
| default group (all members) |
| +---+ +---+ +---+ +---+ +---+ +---+ |
| | A | | B | | C | | D | | E | | F | |
| +---+ +---+ +---+ +---+ +---+ +---+ |
| |
| gpu group highmem group |
| +-----------+ +-----------+ |
| | A | | B | | D | | E | |
| +---+ +---+ +---+ +---+ |
| |
+-----------------------------------------------+
Members A and B are in: default + gpu
Members D and E are in: default + highmem
Members C and F are in: default only
### 2. Why Use Cluster Groups?
Without cluster groups, Incus treats all cluster members equally when deciding where to place a new instance. This works well for homogeneous clusters, but real-world clusters often have servers with different hardware capabilities or different intended roles. Cluster groups solve this by giving you control over *where* workloads land.
Cluster groups enable:
- **Hardware-aware placement** — Direct GPU workloads to GPU-equipped servers, memory-intensive workloads to high-memory servers, or I/O-heavy workloads to SSD-backed servers.
- **Environment separation** — Keep production, staging, and development workloads on different physical servers.
- **Multi-tenancy** — Restrict specific Incus projects to only use certain cluster groups, preventing one tenant's workloads from landing on another tenant's servers.
- **Evacuation safety** — When a server is evacuated for maintenance, Incus respects cluster group constraints and only moves instances to other servers in the same group.
- **CPU compatibility for VM live migration** — Incus computes a CPU baseline per cluster group. Grouping servers by CPU generation ensures reliable live migration within the group.
### 3. The Default Cluster Group
Every Incus cluster has a built-in cluster group called **default**. This group is created automatically when the cluster is initialised and has special properties:
- All new cluster members are **automatically added** to the default group when they join the cluster.
- The default group **cannot be deleted**. Attempting to delete it will result in an error.
- The default group **cannot be renamed**.
- If no target group is specified when launching an instance, Incus selects from all eligible members — which typically means all members in the default group (assuming they have not been removed from it).
**Note:** A member can be removed from the default group by assigning it exclusively to other groups. For example, assigning a member to only the `gpu` group will remove it from the `default` group. If you want a member in both, you must explicitly include both groups when assigning (e.g., `default` and `gpu`).
### 4. Instance Placement & Scheduling
Cluster groups directly influence where new instances are created. Understanding how placement works is key to using cluster groups effectively.
#### 4.1 Targeting a Cluster Group
When creating or moving an instance, you can target a specific cluster group using the `@` prefix. For example, on the command line:
```
`# Launch an instance on any member of the "gpu" group
incus launch images:debian/12 my-instance --target=@gpu
# Move an existing instance to the "highmem" group
incus move my-instance --target=@highmem`
```
In AETHER, when you create an instance and select a target, the cluster group constraint is applied the same way via the API.
When an instance is launched targeting a cluster group, Incus records that constraint in the instance's configuration as `volatile.cluster.group`. This value is used later during evacuation and automatic healing to ensure the instance stays within its designated group.
#### 4.2 Scheduler Modes
Each cluster member has a setting called scheduler.instance that controls how it participates in instance placement. This setting has three possible values:
|
Mode |
Untargeted Launch
(no --target) |
Targeted to
Specific Member |
Targeted to
Cluster Group |
|
**all** (default) |
Yes — participates in global scheduling |
Yes |
Yes |
|
**group** |
No — excluded from untargeted placement |
Yes |
Yes |
|
**manual** |
No |
Yes |
No |
**Tip:** Use `scheduler.instance=group` on servers that should *only* receive instances when explicitly targeted by group name or member name. This is useful for dedicated-purpose servers (e.g., GPU nodes) that should not receive general workloads.
#### 4.3 How Incus Selects a Member
When you target a cluster group, Incus selects the specific member within that group using the following logic:
- **Filter** — Only members in the targeted group whose `scheduler.instance` is set to `all` or `group` are considered.
- **Rank** — Among the eligible members, the one with the **lowest number of instances** is selected.
- **Tie-break** — If multiple members have the same instance count, one is chosen at random.
This simple load-balancing approach ensures that instances are spread evenly across the servers in a group.
### 5. Evacuation & Cluster Groups
Cluster groups play an important role during **member evacuation** (taking a server offline for maintenance). When a cluster member is evacuated:
- Incus reads the `volatile.cluster.group` value from each instance on the evacuated member.
- For instances that have a cluster group constraint, Incus **only migrates them to other members within the same group**.
- For instances without a group constraint, Incus selects from all eligible members in the cluster.
This behaviour ensures that your placement decisions are preserved during maintenance. For example, if you placed a GPU workload on the `gpu` group, evacuating one GPU server will move the workload to *another* GPU server — not to a server without a GPU.
**Important:** For evacuation to succeed with group-constrained instances, there must be at least one other online member in the same cluster group. If a group has only a single member and that member is evacuated, the group-constrained instances will have nowhere to go, and evacuation will fail for those instances. Always plan your groups with redundancy in mind.
The same logic applies to **automatic healing**. If `cluster.healing_threshold` is configured and a member goes offline, Incus will automatically evacuate its instances, respecting cluster group constraints.
### 6. Cluster Group Configuration
Cluster groups can carry configuration keys that influence how Incus behaves for members within that group. The available configuration keys are:
|
Configuration Key |
Description |
|
instances.vm.cpu.<ARCH>.baseline |
Sets the CPU model baseline for virtual machines in this group. The `<ARCH>` placeholder is the CPU architecture (e.g., `x86_64`).
Common values:
- **Not set** (default) — Incus automatically computes the lowest common denominator of CPU features across all members in the group. This is the safest option for mixed-CPU groups.
- **host** — Expose the full host CPU to VMs. Only use this if *all* members in the group have identical CPUs. Provides the best VM performance.
- **A specific CPU model** (e.g., `EPYC-v2`, `qemu64`, `kvm64`) — Pin the baseline to a specific model.
Example: `instances.vm.cpu.x86_64.baseline=host`
|
|
instances.vm.cpu.<ARCH>.flags |
Comma-separated CPU flags to add or remove. Prefix a flag with `-` to remove it.
Use `auto` to let Incus automatically detect common CPU flags across all group members.
Example: `instances.vm.cpu.x86_64.flags=auto`
|
|
user.* |
Free-form custom key-value pairs for your own organisational purposes. These are not interpreted by Incus but can be useful for documentation or tooling. |
**Why CPU baselines matter:** For live migration of virtual machines to work, the target server must present at least the same CPU features as the source. If servers in a group have different CPU generations, Incus computes a baseline that represents the lowest common set of features. By grouping servers with identical CPUs together, you can safely use `baseline=host` and give VMs access to the full CPU feature set, which yields better performance.
### 7. Common Use Cases
#### Hardware-Based Grouping
The most common use case. Group servers by their distinguishing hardware characteristics:
|
Group Name |
Purpose |
|
`gpu` |
Servers with GPU accelerators for AI/ML, rendering, or video encoding workloads |
|
`highmem` |
Servers with large amounts of RAM for database or in-memory processing workloads |
|
`ssd` |
Servers with fast SSD/NVMe storage for I/O-intensive workloads |
|
`arm64` |
Servers with ARM processors (if the cluster has mixed architectures) |
#### Environment Separation
Keep different environments on separate physical hardware:
|
Group Name |
Purpose |
|
`production` |
Servers reserved for production workloads with guaranteed resources |
|
`staging` |
Servers for pre-production testing and validation |
|
`development` |
Servers for developer sandboxes and experimentation |
#### Multi-Tenancy via Project Restrictions
Incus projects can be restricted to only use specific cluster groups. This is a powerful mechanism for multi-tenant environments:
- Set `restricted=true` on a project.
- Set `restricted.cluster.groups=groupname1,groupname2` to limit which groups the project's instances can be placed on.
This ensures that one tenant's instances can never land on another tenant's dedicated servers, even by accident.
#### CPU Generation Grouping for VM Live Migration
For clusters that run virtual machines and rely on live migration:
- Create one group per CPU generation (e.g., `epyc-gen3`, `epyc-gen4`).
- Set `instances.vm.cpu.x86_64.baseline=host` on each group (since all members have identical CPUs).
- VMs within each group can be live-migrated freely with full CPU feature access.
### 8. Table Columns Explained
The Cluster Groups table in AETHER displays the following information for each group:
|
Column |
Description |
|
**Name** |
The unique name of the cluster group. This is set at creation time and cannot be changed afterwards. The `default` group is always present. |
|
**Description** |
An optional human-readable description of the group's purpose. Shows `-` if no description has been set. Can be modified via the Edit modal. |
|
**Members** |
The cluster members (servers) that belong to this group, displayed as individual badges. A member's name corresponds to its hostname in the cluster. If a group has no members, this column will be empty. |
|
**Member Count** |
The total number of cluster members in this group, shown as a numeric value. Useful for quickly gauging group size without counting individual badges. |
|
**Actions** |
Action buttons for managing the group:
- **Edit** — Opens the edit modal to modify the group's description and member list.
- **Delete** — Deletes the group (with confirmation). Not available for the `default` group.
Users with the **Auditor** role will not see action buttons, as auditors have read-only access.
|
### 9. Creating a Cluster Group
Click the **Create Group** button to open the creation form. The form has the following fields:
|
Field |
Required |
Description |
|
**Group Name** |
Yes |
A unique name for the new cluster group. Choose a descriptive, lowercase name that reflects the group's purpose (e.g., `gpu`, `production`, `highmem`). The name cannot be changed after creation. |
|
**Description** |
No |
An optional text description explaining what the group is for. This is displayed in the table and helps other administrators understand the group's purpose. |
|
**Members** |
No |
A list of cluster members to add to the group, presented as checkboxes. You can select zero or more members. Members can also be added or removed later by editing the group. |
**Note:** Creating a cluster group with members does *not* remove those members from other groups they already belong to. A member can belong to multiple groups simultaneously. The member checkboxes show all cluster members available for selection.
### 10. Editing a Cluster Group
Click the **Edit** button on any group row to open the edit modal. You can modify:
- **Description** — Update or clear the group description.
- **Members** — Add or remove cluster members using the checkboxes. Members that are currently in the group will be pre-selected.
**Important:** The group **name cannot be changed** after creation. If you need a different name, you must delete the group and create a new one with the desired name.
When you save changes to a group's membership:
- **Adding a member** to the group does not remove it from any other groups it belongs to.
- **Removing a member** from the group does not affect its membership in other groups.
- The changes take effect immediately. New instances targeted to the group will use the updated member list.
- Existing instances already running on a member are **not affected** by group membership changes.
### 11. Deleting a Cluster Group
Click the **Delete** button on a group row to remove it. A confirmation prompt will appear before deletion proceeds.
When a cluster group is deleted:
- The group is removed from the cluster.
- Members that belonged to the deleted group are **not removed from the cluster** — they simply lose that group membership. They remain in any other groups they belong to.
- If a member was *only* in the deleted group (and no other groups), it will have no group membership until reassigned.
- Existing running instances are not affected.
**Warning:** The `default` cluster group **cannot be deleted**. The Delete button is not shown for the default group, and the operation is also protected server-side.
**Caution:** If instances have `volatile.cluster.group` referencing a group that has been deleted, future evacuations of those instances may not behave as expected, since the group constraint can no longer be resolved. Ensure that no instances depend on a group before deleting it.
### 12. Best Practices
#### Naming Conventions
- Use **short, descriptive, lowercase** names (e.g., `gpu`, `highmem`, `production`).
- Avoid spaces and special characters. Stick to letters, numbers, and hyphens.
- Include context in the description field (e.g., "Servers with NVIDIA A100 GPUs in rack 3").
#### Group Sizing for Redundancy
- Always have **at least two members** in any group that hosts production workloads. If a group has only one member and that member needs maintenance, there is nowhere to evacuate its group-constrained instances to.
- Consider capacity planning: if one member is evacuated, the remaining members in the group must have enough resources to absorb the migrated workloads.
#### Keep Members in the Default Group Unless There Is a Reason Not To
- When a member is only assigned to a custom group and removed from the default group, it will not receive untargeted instances (instances launched without `--target`).
- Only remove a member from the default group if you specifically want it excluded from general-purpose scheduling.
#### Use `scheduler.instance=group` for Dedicated Servers
- If certain servers should **only** receive instances when explicitly targeted (by member name or group name), set `scheduler.instance=group` on those members.
- This prevents dedicated hardware (e.g., GPU nodes) from being filled with general-purpose workloads launched without a target.
#### Plan Groups Around Evacuation
- When planning your cluster groups, think about what happens during maintenance.
- Instances launched with a group target will only be evacuated to other members in the same group.
- Ensure each group has sufficient members and spare capacity to handle a single-member evacuation.
#### Group by CPU Generation for VM Workloads
- If your cluster has servers with different CPU models and you run virtual machines, create one cluster group per CPU generation.
- Set `instances.vm.cpu.x86_64.baseline=host` on groups where all members have identical CPUs. This gives VMs the best performance and enables reliable live migration.
- For mixed-CPU groups, leave the baseline unset and let Incus compute the lowest common denominator automatically.
#### Use Project Restrictions for Multi-Tenancy
- When serving multiple teams or tenants, create dedicated cluster groups for each and restrict their Incus projects to only use those groups.
- Set `restricted=true` and `restricted.cluster.groups=<group>` on each project to enforce isolation.
---
## Incus – Storage Pools
*Source: `/static/help_incus_storage_pools.html`*
AETHER INCUS Storage Pools Guide
## AETHER INCUS Storage Pools Guide
This guide explains what Incus storage pools are, the available storage drivers, and how to create and manage storage pools in AETHER Infrastructure Management.
#### Table of Contents
- 1. What Are Storage Pools?
- 2. How Pools Relate to Volumes
- 3. Storage Drivers
-
- 3.1 Directory (dir)
- 3.2 Btrfs
- 3.3 ZFS
- 3.4 LVM
- 3.5 Ceph RBD (ceph)
- 3.6 CephFS
- 3.7 Ceph Object (cephobject)
- 3.8 LVM Cluster (lvmcluster)
- 3.9 LINSTOR
- 3.10 TrueNAS
- 4. Driver Feature Comparison
- 5. Local vs Shared Storage
- 6. Storage Pools in a Cluster
- 7. Table Columns Explained
- 8. Creating a Storage Pool
- 9. The Source Field
- 10. Viewing and Editing a Pool
- 11. Deleting a Storage Pool
- 12. Best Practices
### 1. What Are Storage Pools?
A **storage pool** is a managed storage unit in Incus that wraps a specific storage technology (called a *driver* or *backend*) and provides a unified interface for storing instance data, images, and custom volumes. Think of a storage pool as a dedicated container for all the disk data that Incus manages.
When you first initialise Incus, you are prompted to create an initial storage pool. This pool becomes the place where instance root filesystems, cached images, and snapshots are stored. You can create additional pools later — using the same or different drivers — to separate workloads, use different hardware, or support different use cases.
Key characteristics of storage pools:
- Each pool uses exactly **one storage driver** (e.g., ZFS, Btrfs, LVM, Ceph).
- A pool has a **name**, an optional **description**, and driver-specific **configuration**.
- All instance data (root disks, snapshots) and cached images live inside a pool.
- You can have **multiple pools** on a single server — for example, a fast SSD pool for databases and a large HDD pool for archival.
- In a cluster, every member must have all the same named pools defined.
### 2. How Pools Relate to Volumes
Storage **volumes** live inside storage pools. A pool is the container; volumes are the individual storage units within it.
|
Volume Type |
Created By |
Purpose |
Lifecycle |
|
**Container / VM** |
Automatically when you launch an instance |
Holds the instance root filesystem |
Deleted when the instance is removed |
|
**Image** |
Automatically when Incus unpacks an image into a pool |
Cached copy of the image for fast instance cloning |
Auto-cleaned after the configured expiry period (default: 10 days) |
|
**Custom** |
Manually by the user |
Independent data storage that can be attached to instances |
Persists until explicitly deleted |
When you launch an instance, Incus creates a root disk volume in the pool specified by the instance's profile (typically the `default` profile). Custom volumes can be created separately and attached to one or more instances as additional storage.
**Volume content types:** Volumes can be *filesystem* (default, shareable between instances), *block* (raw block devices for VMs), or *ISO* (read-only, for VM optical drives). The content type is set at creation and cannot be changed.
### 3. Storage Drivers
Incus supports ten storage drivers. Each driver wraps a different storage technology with its own strengths and trade-offs. The driver you choose determines the features available to you (snapshots, cloning speed, compression, live migration support, etc.).
##### 3.1 Directory (dir) Local
The simplest backend. Stores everything as regular files and directories on the host filesystem. No additional packages or setup required.
**Advantages:**
- Trivial to set up — works out of the box on any Linux filesystem
- Direct file accessibility on disk for inspection and debugging
- Supports storage quotas when on ext4 or XFS with project quotas enabled
**Limitations:**
- Significantly slower than all other drivers
- No copy-on-write — every snapshot and clone requires a full copy
- No instant cloning — creating instances from images requires full extraction every time
- No optimised image storage or transfer
**When to use:** Testing, development, or quick-start environments where performance is not a concern. Serves as a fallback when no other driver is available.
**Source:** Path to an existing directory (e.g., `/data/incus`). Defaults to `/var/lib/incus/storage-pools/<pool_name>`.
##### 3.2 Btrfs Local
A Linux copy-on-write filesystem with built-in volume management, subvolume support, snapshots, compression, and checksums. Creates one subvolume per instance, image, and snapshot.
**Advantages:**
- Copy-on-write with instant cloning and snapshots
- Optimised image storage, instance creation, and transfers (native send/receive)
- Compression support via mount options
- Can share the filesystem with the host if it already runs Btrfs
- Supports nested Incus/container environments
**Limitations:**
- Quotas via qgroups are problematic — subvolumes can escape parent quotas
- Less mature RAID implementation compared to ZFS
**Warning:** Never use VMs with Btrfs storage pools. The driver creates large files for VM storage rather than native block devices, causing inefficiency and snapshot issues.
**When to use:** Container-only workloads where you want good performance with a simpler setup than ZFS. Good choice if the host already runs Btrfs.
**Source:** Block device (e.g., `/dev/sdX`), loop file, or existing Btrfs subvolume path.
##### 3.3 ZFS Local Recommended
A combined physical volume manager and filesystem with strong data protection through checksums, copy-on-write, compression, and built-in RAID support. Requires the `zfsutils-linux` package.
**Advantages:**
- Copy-on-write with instant snapshots and clones
- Built-in data integrity via checksums
- Compression enabled by default on new pools
- RAID management (mirror, raidz, raidz2, raidz3)
- Optimised image storage, instance creation, and transfers (native send/receive)
- Trimming support for SSD optimisation
- Can delegate datasets to containers (ZFS 2.2+)
- Works well for both containers and VMs
**Limitations:**
- Cannot restore from snapshots other than the latest one (must create new instances from older snapshots)
- I/O quotas have minimal effectiveness (ZFS bypasses Linux VFS API)
- Higher memory usage than simpler drivers
**When to use:** Production workloads. The official Incus documentation states: *"The two best options for use with Incus are ZFS and Btrfs. They have similar functionalities, but ZFS is more reliable."* ZFS is the best all-round choice for both containers and VMs.
**Source:** Block device, loop file, existing ZFS dataset (e.g., `my-tank/slice`), or zpool name.
##### 3.4 LVM Local
Linux Logical Volume Manager — a storage management framework that virtualises physical storage into logical volumes. Uses thin provisioning by default for efficient space usage. Requires the `lvm2` package.
**Advantages:**
- Thin provisioning by default — allows over-commitment of storage
- Efficient snapshots with thin pools
- Mature, battle-tested technology well understood by Linux administrators
- Can use dedicated disks or partitions
**Limitations:**
- Without thin pools, performance is similar to the dir driver
- No optimised image or volume transfer (no native send/receive)
- Not usable inside a container
**When to use:** When you need a well-known, stable backend with thin provisioning. Good for environments already using LVM. Always keep thin pools enabled (the default).
**Source:** Block device, loop file, or existing LVM volume group name (e.g., `my-vg`).
##### 3.5 Ceph RBD (ceph) Shared
Ceph RADOS Block Device — distributed block storage from a Ceph cluster. Provides shared storage accessible from all cluster members. Requires a pre-existing Ceph cluster.
**Advantages:**
- Shared storage — all cluster members access the same pool and volumes
- Thin provisioning and copy-on-write
- Highly scalable, fault-tolerant with no single point of failure
- Optimised image storage, instance creation, and transfers
- Enables live migration of instances between cluster members
**Limitations:**
- Requires a pre-existing Ceph cluster (Incus does not set up Ceph)
- Remote access may be slower than local storage
- Incus assumes full control over the OSD pool — unauthorised entities will be deleted
- Not usable inside a container
**When to use:** Clustered Incus deployments where you need shared storage across all members. The go-to choice for clusters that need live migration and high availability.
**Source:** Existing Ceph OSD pool name (auto-created if missing).
##### 3.6 CephFS Shared
Ceph's distributed POSIX-compliant filesystem. Provides shared filesystem volumes accessible from all cluster members. Requires a pre-existing Ceph cluster.
**Advantages:**
- Shared filesystem storage with cluster-wide access
- POSIX-compliant distributed filesystem
- Supports snapshots and can auto-create missing filesystem and OSD pools
- Filesystem volumes can be shared between instances
**Limitations:**
- Only supports custom storage volumes with filesystem content type
- Cannot be used for instance root disks — use alongside Ceph RBD
- Requires pre-existing Ceph cluster
**When to use:** When you need shared filesystem volumes accessible by multiple instances across a cluster. Typically used alongside a Ceph RBD pool (which handles instance root disks).
**Source:** Existing CephFS filesystem name.
##### 3.7 Ceph Object (cephobject) Shared
Ceph Object Gateway (RADOS Gateway) providing S3-compatible object storage. Only supports storage buckets — not regular volumes or instance disks.
**Advantages:**
- S3-compatible object storage accessible via HTTP/HTTPS
- Cluster-wide access without synchronisation
- Leverages existing Ceph infrastructure
**Limitations:**
- Only supports storage buckets — not for volumes or instance disks
- Requires a pre-existing Ceph cluster with a configured RADOS Gateway
**When to use:** When you need S3-compatible object storage for applications running in Incus instances. Typically used alongside Ceph RBD and/or CephFS.
##### 3.8 LVM Cluster (lvmcluster) Shared
A clustered variant of LVM designed for Incus clusters. Uses `lvmlockd` and `sanlock` for distributed locking on shared block devices (FibreChannel LUNs, NVMEoF/NVMEoTCP disks, iSCSI drives).
**Advantages:**
- Shared block storage across all cluster nodes via shared block devices
- Enables live VM migration between cluster members
- Distributed locking prevents concurrent modification conflicts
**Limitations:**
- Thin provisioning is incompatible with clustered LVM — expect higher disk usage
- Requires significant infrastructure setup: shared block device, lvmlockd, sanlock
- More complex to set up than regular LVM
**When to use:** When you have a shared SAN/NVMe fabric and need shared block storage across cluster members without Ceph.
##### 3.9 LINSTOR Shared
Open-source software-defined storage using DRBD kernel-space replication. Manages ZFS or LVM block devices and replicates them across nodes. Requires a pre-existing LINSTOR cluster.
**Advantages:**
- Kernel-space replication for high performance
- Hybrid storage: diskful replicas for local low-latency I/O, diskless replicas for network-based I/O
- Configurable replica count (default: 2 diskful replicas)
- Copy-on-write, instant cloning, and optimised snapshots
**Limitations:**
- Requires a pre-existing LINSTOR cluster with controllers and satellites
- Incus node names must match LINSTOR satellite names
- Cannot restore from non-latest snapshots directly
**When to use:** When you need replicated block storage across cluster nodes without the complexity of Ceph. Good for HA setups with 2–3 node clusters.
##### 3.10 TrueNAS Shared
Remote block storage via a TrueNAS server, accessed over iSCSI. Operates similarly to ZFS but remotely. Requires a pre-existing TrueNAS server and the `truenas_incus_ctl` tool.
**Advantages:**
- Remote storage enabling cluster-wide shared access
- ZFS-based on the TrueNAS side (snapshots, cloning, efficient image handling)
- Enables live VM migration between cluster members
- Does not require nested ZFS on the Incus host
**Limitations:**
- Requires a pre-existing TrueNAS server and `open-iscsi` on all Incus nodes
- Remote iSCSI access introduces latency compared to local storage
**When to use:** When you have a TrueNAS storage appliance and want to use it as shared storage for an Incus cluster without the complexity of Ceph.
### 4. Driver Feature Comparison
This table summarises the key capabilities of each driver:
|
Feature |
dir |
btrfs |
zfs |
lvm |
ceph |
cephfs |
linstor |
truenas |
|
Copy-on-write |
No |
Yes |
Yes |
Yes |
Yes |
Yes |
Yes |
Yes |
|
Instant cloning |
No |
Yes |
Yes |
Yes |
Yes |
Yes |
Yes |
Yes |
|
Optimised image storage |
No |
Yes |
Yes |
Yes |
Yes |
N/A |
Yes |
Yes |
|
Optimised transfers |
No |
Yes |
Yes |
No |
Yes |
N/A |
No |
No |
|
Restore older snapshots |
Yes |
Yes |
No |
Yes |
Yes |
Yes |
No |
Yes |
|
Storage quotas |
Yes* |
Yes |
Yes |
Yes |
Yes |
Yes |
Yes |
Yes |
|
Shared across cluster |
No |
No |
No |
No |
Yes |
Yes |
Yes |
Yes |
|
VM support |
Yes |
No |
Yes |
Yes |
Yes |
No |
Yes |
Yes |
|
Object storage (buckets) |
Yes |
Yes |
Yes |
Yes |
No |
No |
No |
No |
* dir driver requires ext4 or XFS with project quotas enabled for storage quotas.
### 5. Local vs Shared Storage
This is one of the most important distinctions for cluster administrators:
#### Local Storage Local
Drivers: **dir**, **btrfs**, **lvm**, **zfs**
- Each cluster member has its own **independent copy** of the pool on its local disk.
- Volumes created in local pools exist **only on the member** where they were created.
- Instances using local storage **cannot be live-migrated** between members (they require stop-and-migrate).
- Each member can have different `source`, `size`, and driver-specific paths for the same named pool.
#### Shared/Remote Storage Shared
Drivers: **ceph**, **cephfs**, **cephobject**, **lvmcluster**, **linstor**, **truenas**
- All cluster members access the **same storage** pool with the same volumes.
- Volumes are **accessible from any member** without synchronisation.
- Enables **live migration** of instances between cluster members.
- No member-specific source configuration needed — all members point to the same external storage system.
**Tip:** Many production clusters use a combination of both — a local ZFS pool for fast instance storage and a Ceph pool for instances that need live migration or high availability.
### 6. Storage Pools in a Cluster
In an Incus cluster, storage pools have additional considerations:
#### All Members Must Have the Same Named Pools
Every cluster member must have all the same storage pools defined. The pool name must be identical across all members, but certain configuration keys are allowed to differ between members:
- source — where the pool data lives on each member
- size — pool size (for loop-backed configurations)
- zfs.pool_name — the ZFS pool/dataset name
- lvm.vg_name — the LVM volume group name
- lvm.thinpool_name — the LVM thin pool name
All other configuration must be identical across members.
#### How Pool Creation Works in a Cluster
The creation process differs depending on whether the storage driver is **local** or **shared**:
##### Shared Drivers (Ceph RBD, CephFS, Ceph Object)
For shared/remote storage drivers, the pool exists centrally (in the Ceph cluster) and is accessible from all members. A single creation command is sufficient — there is no need to configure each member individually. AETHER's Create Pool form handles this correctly.
##### Local Drivers (dir, Btrfs, ZFS, LVM)
For local storage drivers, the pool must be configured on **each cluster member individually**, because each member has its own local disk and may need different settings (different `source` path, different `size`, etc.). Incus requires a two-step process:
- **Define per-member configuration** — For each cluster member, specify the member-specific options (source, size, etc.) using `--target=<member>`. During this phase, the pool is in **Pending** state.
- **Finalise across the cluster** — Once all members have been configured, a final creation command without `--target` instantiates the pool on all members simultaneously. The pool transitions to **Created** state.
For example, creating a ZFS pool across a three-member cluster via the CLI:
```
`# Step 1: Define per-member configuration
incus storage create my-pool zfs source=/dev/sdX size=10GiB --target=server1
incus storage create my-pool zfs source=/dev/sdX size=15GiB --target=server2
incus storage create my-pool zfs source=/dev/sdY size=10GiB --target=server3
# Step 2: Finalise (creates the pool on all members)
incus storage create my-pool zfs`
```
**Current AETHER limitation:** The Create Pool form in AETHER currently creates the pool targeting a **single cluster member**. For local storage drivers, this means the pool will only exist on that one member. To complete the setup across all members, you will need to use the Incus CLI to define the pool on the remaining members and then finalise it. This is a known limitation that may be addressed in a future AETHER release.
#### Understanding the Pending Status
If you see a storage pool with a **Pending** status, it means the pool has been defined on one or more cluster members but has **not yet been finalised** across the entire cluster. This typically happens when:
- A local storage pool was created but not all members have been configured yet.
- The final creation step (without `--target`) has not been run.
A Pending pool cannot be used to create volumes or launch instances until it reaches the **Created** state.
#### Joining New Members
When a new member joins an existing cluster, it must provide configuration for all storage pools that already exist in the cluster. The join process prompts for member-specific values (source, size, etc.) so that each pool can be set up on the new member's local storage.
### 7. Table Columns Explained
The Storage Pools table in AETHER displays the following information for each pool:
|
Column |
Description |
|
**Name** |
The unique name of the storage pool. This is set at creation time and cannot be changed afterwards. Common names include `default`, `local`, `fast-ssd`, or `ceph-pool`. |
|
**Driver** |
The storage backend in use. Possible values:
- `dir` — Directory
- `btrfs` — Btrfs filesystem
- `zfs` — ZFS
- `lvm` — LVM (Logical Volume Manager)
- `lvmcluster` — Clustered LVM
- `ceph` — Ceph RBD (block storage)
- `cephfs` — CephFS (filesystem)
- `cephobject` — Ceph Object (S3-compatible)
- `linstor` — LINSTOR/DRBD
- `truenas` — TrueNAS (iSCSI)
|
|
**Size** |
A visual progress bar showing storage usage. Displays: **used / total (percentage)**.
The progress bar is colour-coded based on usage percentage:
- **Green** — Usage below 75%. Healthy, plenty of free space.
- **Amber** — Usage between 75% and 90%. Consider freeing space or expanding the pool.
- **Red** — Usage above 90%. Critical — the pool is nearly full. Immediate action recommended.
Example: `45.2 GiB / 100.0 GiB (45%)`
**Note:** Some drivers (such as Ceph) may not report size information in the same way as local drivers. If size data is unavailable, the column may appear empty or show a dash.
|
|
**Description** |
An optional human-readable description of the pool's purpose. Shows `-` if no description has been set. Can be modified via the View/Edit modal. |
|
**Used By** |
A numeric count of **all API objects that reference this storage pool**. This includes:
- **Profiles** — Any profile with a root disk device pointing to this pool (the `default` profile counts as 1).
- **Instances** — Every container and VM whose root disk volume resides in this pool.
- **Custom volumes** — User-created volumes in the pool.
- **Image volumes** — Cached image volumes that have been unpacked into the pool.
For example, a pool showing `Used By: 50` might have 1 profile, 5 cached images, 42 instances, and 2 custom volumes.
A pool **cannot be deleted** while its Used By count is greater than zero.
|
|
**Status** |
The current operational state of the pool, shown as a coloured badge:
- **Created** (green) — The pool is fully created and operational. This is the normal, healthy state.
- **Pending** — The pool has been defined on some cluster members but has not yet been finalised across the entire cluster. Awaiting the final creation step.
- **Errored** — The pool is in an error state. Creation may have failed on one or more cluster members, or the underlying storage became inaccessible.
- **Unavailable** — The pool failed to initialise. The underlying storage may be missing, misconfigured, or disconnected.
- **Unknown** — The pool is in an indeterminate state, possibly indicating an internal error.
|
|
**Actions** |
Action buttons for managing the pool:
- **View/Edit** — Opens the detail modal where you can view full pool properties, modify the description, and edit configuration key-value pairs.
- **Delete** — Deletes the pool (with confirmation). This button **only appears when Used By is 0**. You cannot delete a pool that is still in use.
|
### 8. Creating a Storage Pool
Click the **Create Pool** button to open the creation form. The form has the following fields:
|
Field |
Required |
Description |
|
**Name** |
Yes |
A unique name for the new storage pool. Choose a descriptive, lowercase name that reflects the pool's purpose or driver (e.g., `local-zfs`, `fast-ssd`, `ceph-shared`). The name cannot be changed after creation. |
|
**Driver** |
Yes |
The storage backend to use. Select from the dropdown:
- **ZFS** — Recommended for most deployments
- **BTRFS** — Good for container-only workloads
- **Directory** — Simplest, for testing only
- **LVM** — Traditional Linux volume management
- **Ceph RBD** — Shared block storage (requires Ceph cluster)
- **CephFS** — Shared filesystem (requires Ceph cluster)
- **Ceph Object** — S3-compatible buckets (requires Ceph cluster)
Selecting a driver reveals driver-specific configuration options below.
|
|
**Description** |
No |
An optional text description explaining what the pool is for. Helps other administrators understand the pool's purpose. |
#### Driver-Specific Options
After selecting a driver, additional configuration fields appear. The most common options per driver are:
##### ZFS Options
| Option | Description |
|
source |
Path to a block device (e.g., `/dev/sdb`), a loop file, or an existing ZFS dataset/pool name. If left empty, Incus creates a loop file automatically. |
|
size |
Size of the loop file (e.g., `50GiB`). Only applies when no block device is specified as source. Defaults to 20% of free disk space (min 5 GiB, max 30 GiB). |
|
zfs.pool_name |
Name of the ZFS pool or dataset to use. Defaults to the Incus pool name. |
|
source.wipe |
Wipe the block device before using it. Use with caution — this destroys existing data. |
|
zfs.clone_copy |
Whether to use ZFS lightweight clones. Options: `true` (default, uses clones), `false` (full copies), `rebase` (optimise clone chains). |
|
zfs.export |
Whether to export the ZFS pool on unmount. Default: `true`. |
##### Btrfs Options
| Option | Description |
|
source |
Path to a block device or existing Btrfs subvolume. If left empty, Incus creates a loop file. |
|
size |
Size of the loop file (e.g., `50GiB`). Only applies when no block device is specified. |
|
btrfs.mount_options |
Additional mount options for the Btrfs filesystem (e.g., `compress=zstd`). |
|
source.wipe |
Wipe the block device before use. |
##### Directory Options
| Option | Description |
|
source |
Path to an existing directory. Defaults to `/var/lib/incus/storage-pools/<name>`. |
|
rsync.bwlimit |
Bandwidth limit for rsync operations (in KiB/s). Empty means no limit. |
|
rsync.compression |
Whether to use compression during rsync transfers. Default: `true`. |
##### LVM Options
| Option | Description |
|
source |
Path to a block device, loop file, or existing LVM volume group name. |
|
size |
Size of the loop file. Only applies when no block device or VG is specified. |
|
lvm.vg_name |
Name of the LVM volume group. Defaults to the Incus pool name. |
|
lvm.thinpool_name |
Name of the LVM thin pool. Default: `IncusThinPool`. |
|
lvm.use_thinpool |
Whether to use a thin pool. Default: `true`. Strongly recommended to keep enabled. |
|
lvm.vg.force_reuse |
Force using an existing non-empty volume group. |
|
source.wipe |
Wipe the block device before use. |
##### Ceph RBD Options
| Option | Description |
|
source |
Existing Ceph OSD pool name. Auto-created if it does not exist. |
|
ceph.cluster_name |
Name of the Ceph cluster. Default: `ceph`. |
|
ceph.user.name |
Ceph user for authentication. Default: `admin`. |
|
ceph.osd.pool_name |
Name of the OSD pool for RBD images. |
|
ceph.osd.pg_num |
Number of placement groups for the OSD pool. Default: `32`. |
|
ceph.osd.data_pool_name |
Optional data pool name for erasure-coded pools. |
|
ceph.rbd.clone_copy |
Whether to use RBD lightweight clones. Default: `true`. |
|
ceph.rbd.features |
Comma-separated list of RBD features to enable. |
##### CephFS Options
| Option | Description |
|
source |
Existing CephFS filesystem name. |
|
cephfs.cluster_name |
Name of the Ceph cluster. Default: `ceph`. |
|
cephfs.user.name |
Ceph user for authentication. Default: `admin`. |
|
cephfs.path |
Base CephFS path for the pool. |
|
cephfs.create_missing |
Whether to create missing CephFS filesystem and pools. Default: `false`. |
|
cephfs.data_pool |
Data OSD pool name. |
|
cephfs.meta_pool |
Metadata OSD pool name. |
|
cephfs.fscache |
Whether to enable kernel fscache. Default: `false`. |
##### Ceph Object Options
| Option | Description |
|
cephobject.radosgw.endpoint |
The RADOS Gateway endpoint URL (required). E.g., `https://radosgw.example.com`. |
|
cephobject.cluster_name |
Name of the Ceph cluster. Default: `ceph`. |
|
cephobject.user.name |
Ceph user for authentication. Default: `admin`. |
|
cephobject.radosgw.endpoint_cert_file |
Path to a custom CA certificate for the RADOS Gateway endpoint. |
|
cephobject.bucket_name_prefix |
Prefix for bucket names created by Incus. |
#### Additional Configuration
The **Additional Configuration** section allows you to add arbitrary key-value pairs to the pool's configuration. Click **+ Add Custom Config** to add a new row. This is useful for setting advanced or driver-specific options that are not shown in the driver-specific options above.
### 9. The Source Field
The source field tells Incus where to find or create the underlying storage. Its meaning varies by driver:
|
Driver |
Source Field Meaning |
Examples |
|
**dir** |
Path to an existing directory |
`/data/incus` |
|
**btrfs** |
Block device, loop file, or existing Btrfs subvolume path |
`/dev/sdb` |
|
**zfs** |
Block device, loop file, existing ZFS dataset, or zpool name |
`/dev/sdb`, `my-tank/incus` |
|
**lvm** |
Block device, loop file, or existing LVM volume group name |
`/dev/sdb`, `my-vg` |
|
**lvmcluster** |
Shared block device accessible from all cluster nodes |
`/dev/shared-lun` |
|
**ceph** |
Ceph OSD pool name (auto-created if missing) |
`incus-pool` |
|
**cephfs** |
Existing CephFS filesystem name |
`my-cephfs` |
|
**cephobject** |
Not applicable — uses `cephobject.radosgw.endpoint` instead |
N/A |
|
**linstor** |
Not applicable — configured via LINSTOR resource groups |
N/A |
|
**truenas** |
Remote ZFS dataset location |
`my-zpool/dataset` |
**Tip:** When no source is specified for drivers that support loop files (btrfs, lvm, zfs), Incus automatically creates a loop file at `/var/lib/incus/disks/<pool_name>.img`. This is convenient for testing but not recommended for production — loop files are slower and cannot be shrunk once grown.
### 10. Viewing and Editing a Pool
Click **View/Edit** on any pool row to open the detail modal. The modal shows:
- **Name** (read-only) — The pool name, which cannot be changed.
- **Driver** (read-only) — The storage backend, which cannot be changed after creation.
- **Status** (read-only) — Current operational state.
- **Storage Usage** (read-only) — Used and total space with percentage.
- **Description** (editable) — Modify the pool description.
- **Locations** (read-only) — Which cluster members have this pool. For shared drivers, shows all members. For local drivers, shows each member that has the pool configured.
- **Used By** (read-only) — List of API resource paths referencing this pool (profiles, instances, volumes, images).
- **Configuration** (editable) — Key-value pairs for the pool's configuration. You can modify existing values, add new keys with **+ Add Config**, or remove keys.
**Important:** Be careful when modifying pool configuration after creation. Changing certain driver-specific keys on a running pool can cause data loss or make the pool inaccessible. Only modify configuration keys that you fully understand.
### 11. Deleting a Storage Pool
A storage pool can only be deleted when its **Used By count is zero**. If the pool is still in use, the Delete button will not appear in the Actions column.
To delete a pool that is in use, you must first remove all references to it:
- Delete or move all instances whose root disks reside in the pool.
- Delete all custom volumes stored in the pool.
- Remove the pool from any profiles that reference it (or point them to a different pool).
- Wait for cached image volumes to expire (or manually delete them).
A confirmation prompt will appear before deletion proceeds.
**Warning:** Deleting a storage pool permanently destroys all data stored within it. For local drivers, the underlying block device, loop file, or directory may also be removed depending on the driver. This action cannot be undone.
### 12. Best Practices
#### Choosing a Storage Driver
- **For production, use ZFS or Btrfs.** The official Incus documentation recommends these two as the best options, noting that ZFS is more reliable.
- **Use ZFS if you run virtual machines.** Btrfs has known issues with VMs and should be avoided for VM workloads.
- **Use Ceph for clusters that need live migration.** Ceph RBD provides shared storage accessible from all cluster members, enabling seamless live migration.
- **Use the dir driver only for testing.** It lacks copy-on-write, instant cloning, and optimised transfers.
- **For LVM, always keep thin pools enabled.** Without thin pools, LVM performance degrades significantly and behaves similarly to the dir driver.
#### Storage Sizing
- **Use dedicated disks or partitions for production.** Loop files are convenient for testing but introduce performance overhead and cannot be shrunk once grown.
- **Plan for image cache space.** Incus caches unpacked images in the pool (kept for 10 days by default). With many images, this can consume significant space.
- **Monitor pool usage regularly.** The Size column's colour-coded progress bar makes it easy to spot pools approaching capacity at a glance.
- **For ZFS, compression is enabled by default** on new pools, which can significantly reduce actual disk usage.
- **For LVM thin pools, monitor carefully.** Over-commitment (total logical volume sizes exceeding physical capacity) is useful but can cause out-of-space conditions if not monitored.
#### Pool Organisation
- **Consider multiple pools for different workload types.** For example, a fast SSD-backed pool for databases and a larger HDD-backed pool for file storage or archival.
- **Use descriptive names and descriptions.** Pool names like `fast-ssd` or `ceph-shared` are more helpful than `pool1` or `pool2`.
- **In clusters, plan shared vs local storage carefully.** Use local pools for performance-sensitive workloads that do not need migration, and shared pools (Ceph, LINSTOR, TrueNAS) for workloads that need high availability.
#### Data Safety
- **Never change driver-specific source configuration on a running pool** unless you fully understand the consequences.
- **Use `source.wipe` with extreme caution.** This option destroys all existing data on the specified block device.
- **Keep backups independent of the storage pool.** Snapshots within a pool protect against accidental deletion but not against pool-level failures (e.g., disk failure without RAID).
---
## Incus – Storage Volumes
*Source: `/static/help_incus_storage_volumes.html`*
AETHER INCUS Storage Volumes Guide
## AETHER INCUS Storage Volumes Guide
This guide explains what Incus storage volumes are, the different volume and content types, and how to create and manage volumes in AETHER Infrastructure Management.
#### Table of Contents
- 1. What Are Storage Volumes?
- 2. Volume Types
- 3. Content Types
- 4. Table Columns Explained
- 5. Filtering Volumes
- 6. Creating a Custom Volume
- 7. Viewing and Editing a Volume
- 8. Attaching Volumes to Instances
- 9. Volume Configuration Options
- 10. Snapshots
- 11. Volumes in a Cluster
- 12. Moving and Copying Volumes
- 13. Exporting and Importing Volumes
- 14. Best Practices
### 1. What Are Storage Volumes?
Storage **volumes** are the individual storage units that live inside **storage pools**. If a storage pool is like a disk, then volumes are like the partitions on that disk. Every piece of data that Incus manages — instance root filesystems, cached images, and user data — is stored as a volume within a pool.
Storage Pool (e.g., "default", ZFS driver)
|
+-- container/web-server-01 (auto-created root volume)
+-- container/db-server-01 (auto-created root volume)
+-- virtual-machine/win-desktop (auto-created root volume)
+-- image/a3f5b8c9d1e2... (auto-created image cache)
+-- custom/shared-data (user-created custom volume)
+-- custom/database-backup (user-created custom volume)
Key facts about storage volumes:
- Every volume belongs to exactly **one storage pool**.
- The pool's storage driver (ZFS, Btrfs, LVM, Ceph, etc.) determines what features are available for volumes (snapshot efficiency, cloning speed, quotas, etc.).
- Every instance needs at least one volume — its **root disk** — which is created automatically when the instance is launched.
- You can create additional **custom volumes** for data that should be managed independently of any instance.
- Pool-level defaults (e.g., `volume.size`, `volume.snapshot.schedule`) apply to all new volumes unless overridden at the volume level.
### 2. Volume Types
Incus has four volume types, each serving a different purpose:
|
Type |
Created By |
Purpose |
Lifecycle |
|
container |
Automatically when you launch a container |
Holds the container's root filesystem |
Tied to the container — deleted when the container is removed |
|
virtual-machine |
Automatically when you launch a VM |
Holds the VM's root disk |
Tied to the VM — deleted when the VM is removed |
|
image |
Automatically when Incus unpacks an image into a pool |
Cached copy of the image for fast instance cloning |
Auto-cleaned after 10 days of non-use (configurable), or manually deleted |
|
custom |
Manually by the administrator |
Independent data storage that can be attached to instances |
Persists until explicitly deleted — survives instance deletion |
#### Instance Volumes (container / virtual-machine)
When you launch an instance, Incus automatically creates a storage volume of the corresponding type in the pool specified by the instance's profile (typically the `default` profile). This volume *is* the instance's root filesystem. You cannot delete it independently — it is removed when the instance is deleted.
#### Image Volumes
When Incus launches an instance from an image for the first time on a given pool, it unpacks the image into an image volume. This cached volume allows subsequent instances from the same image to be created much faster (via copy-on-write cloning on drivers like ZFS and Btrfs). Image volumes are automatically deleted 10 days after their last use, or you can delete them manually to reclaim space.
#### Custom Volumes
Custom volumes are the ones you create and manage yourself. They are completely independent of any instance — if you delete a container that had a custom volume attached, the custom volume remains intact. This makes custom volumes ideal for:
- Database storage that should survive instance recreation
- Shared files accessible by multiple instances
- Application data, user uploads, or logs
- Backup targets or staging areas
**Naming note:** Custom volumes may share the same name as instance volumes. For example, you could have a container named `web01` and a custom volume also named `web01`. To avoid ambiguity, instance volumes are always referenced with their type prefix: `container/web01` or `virtual-machine/web01`. Custom volumes can omit the prefix.
### 3. Content Types
Custom storage volumes have one of three **content types** that determine how the volume stores data and what it can be attached to:
|
Content Type |
Description |
Containers |
VMs |
Shareable |
Created Via |
|
**filesystem** |
A filesystem that gets mounted at a path inside the instance (e.g., `/data`). This is the default content type. |
Yes |
Yes* |
Yes |
Create Volume form |
|
**block** |
A raw block device. The instance manages its own filesystem on the device. |
No |
Yes |
No** |
Create Volume form |
|
**iso** |
A read-only ISO image. Appears as a virtual CD/DVD drive in the VM. |
No |
Yes |
Yes (read-only) |
Custom ISOs tab (import) |
* Filesystem volumes cannot be attached to a *running* VM — the VM must be stopped first, then the volume attached, then the VM started.
** Block volumes should never be attached to more than one VM simultaneously. Concurrent access to a raw block device without a cluster-aware filesystem (like GFS2 or OCFS2) causes data corruption.
**Important:** The content type is set at creation time and **cannot be changed** afterwards. Choose carefully when creating a volume.
### 4. Table Columns Explained
The Storage Volumes table in AETHER displays the following information for each volume:
|
Column |
Description |
|
**Name** |
The name of the volume. For instance volumes (container/virtual-machine), this matches the instance name. For custom volumes, this is the name you chose at creation. |
|
**Type** |
The volume type, displayed as a coloured badge:
- custom — User-created data volume
- container — Root disk of a container instance
- virtual-machine — Root disk of a VM instance
- image — Cached image volume for fast instance cloning
|
|
**Content Type** |
How the volume stores data:
- **filesystem** — A mounted filesystem (default for custom volumes and containers)
- **block** — A raw block device (used by VMs and block custom volumes)
- **iso** — A read-only ISO image (imported via Custom ISOs tab)
|
|
**Pool** |
The name of the storage pool this volume resides in (e.g., `default`, `local-zfs`, `ceph-shared`). A volume always belongs to exactly one pool. |
|
**Used By** |
A numeric count of all entities that reference this volume. This includes:
- **Instances** that have the volume attached as a disk device
- **Profiles** that reference the volume in their device configuration
For instance root volumes (container/virtual-machine type), this is typically `1` (the instance itself). For custom volumes, this is the number of instances the volume is attached to. A value of `0` means the volume is not attached to anything.
|
|
**Location** |
The cluster member where the volume physically resides.
- For **local storage** drivers (dir, btrfs, lvm, zfs): Shows the specific member name (e.g., `cluster1`). The volume only exists on this member.
- For **shared storage** drivers (ceph, cephfs, lvmcluster, linstor, truenas): May be empty or show a shared indicator, since the volume is accessible from all members.
|
|
**Created** |
The date and time when the volume was created, displayed in your local time zone. |
### 5. Filtering Volumes
The volumes table can contain a large number of entries (especially in clusters with many instances). Two filter dropdowns help you narrow down the list:
|
Filter |
Description |
|
**Pool Filter** |
Filter volumes by the storage pool they belong to. Select **All Pools** to see volumes across all pools, or select a specific pool name to show only volumes in that pool. |
|
**Type Filter** |
Filter volumes by their type:
- **All Types** — Show all volumes
- **Custom** — Show only user-created custom volumes
- **Container** — Show only container root disk volumes
- **Virtual Machine** — Show only VM root disk volumes
- **Image** — Show only cached image volumes
|
**Tip:** To quickly find your custom data volumes, set the Type filter to **Custom**. This hides the often-numerous instance root volumes and image cache volumes, showing only the volumes you created manually.
### 6. Creating a Custom Volume
Click the **Create Volume** button to open the creation form. The form has the following fields:
|
Field |
Required |
Description |
|
**Storage Pool** |
Yes |
The pool where the volume will be created. Select from the dropdown of available pools. The pool's driver determines the volume's capabilities (snapshot efficiency, cloning, quotas, etc.). |
|
**Name** |
Yes |
A unique name for the volume within the selected pool. Choose a descriptive name that reflects the volume's purpose (e.g., `webapp-data`, `postgres-db`, `shared-files`). |
|
**Content Type** |
Yes |
How the volume stores data. This **cannot be changed** after creation:
- **Filesystem** (default) — A mounted filesystem. Use this for most purposes: shared data, application files, database storage. Can be attached to both containers and VMs.
- **Block** — A raw block device. Use this when the VM needs to manage its own filesystem (e.g., for a database engine that requires raw device access). Can only be attached to VMs.
|
|
**Size** |
No |
The size quota for the volume (e.g., `10GiB`, `500MiB`). If left empty, the volume has no size limit and can grow to fill the pool. Setting a size is recommended for production to prevent a single volume from consuming all pool space. |
|
**Description** |
No |
An optional text description explaining what the volume is for. Helps other administrators understand the volume's purpose. |
|
**Additional Configuration** |
No |
Arbitrary key-value pairs for advanced configuration. Click **+ Add Config** to add rows. See the Volume Configuration Options section for available keys. |
**Note:** ISO volumes cannot be created through this form. To add an ISO volume, use the **Custom ISOs** sub-tab, which allows you to upload or import ISO files.
### 7. Viewing and Editing a Volume
Click on a volume's **View/Edit** button to open the detail modal. The modal shows:
- **Name** (read-only) — The volume name, which cannot be changed through this interface.
- **Type** (read-only) — The volume type (custom, container, virtual-machine, image).
- **Content Type** (read-only) — filesystem, block, or iso.
- **Pool** (read-only) — The storage pool the volume belongs to.
- **Location** (read-only) — Which cluster member hosts the volume.
- **Created** (read-only) — Creation date and time.
- **Description** (editable for custom volumes) — Modify the volume description.
- **Used By** (read-only) — List of instances and profiles referencing this volume.
- **Configuration** (editable for custom volumes) — Key-value pairs. You can modify existing values, add new keys with **+ Add Config**, or remove keys.
**Note:** Instance root volumes (container/virtual-machine type) and image volumes are shown in read-only mode, since their configuration is managed through the instance or image settings rather than directly.
### 8. Attaching Volumes to Instances
Custom volumes become useful when attached to instances. The attachment rules depend on the volume's content type:
|
Content Type |
Attach to Container? |
Attach to VM? |
Requires Mount Path? |
Shareable? |
|
**filesystem** |
Yes |
Yes (VM must be stopped first) |
Yes (e.g., `/data`) |
Yes — can be attached to multiple instances simultaneously |
|
**block** |
No |
Yes |
No (appears as a block device) |
No — concurrent access causes data corruption |
|
**iso** |
No |
Yes |
No (appears as virtual CD/DVD) |
Yes — read-only, safe for multiple VMs |
#### How Attachment Works
- **Filesystem volumes** are mounted at a specific path inside the instance. For example, attaching a volume at `/data` means the instance sees a new filesystem at that path. Multiple instances can mount the same filesystem volume simultaneously for shared data access.
- **Block volumes** appear as a raw block device (like `/dev/sdb`) inside the VM. The VM's operating system is responsible for creating and managing the filesystem on the device.
- **ISO volumes** appear as a virtual CD/DVD drive in the VM. They are always read-only.
#### I/O Limits
When attaching a volume as a disk device, you can optionally set I/O bandwidth limits:
- limits.read — Read bandwidth limit (e.g., `100MB`)
- limits.write — Write bandwidth limit
- limits.max — Combined read+write limit
These use the Linux `blkio` cgroup controller and apply at the block device level.
**Important:** When sharing a filesystem volume between instances that use different UID/GID maps (common with isolated containers), you may need to enable security.shifted on the volume. This uses the kernel's VFS idmapped mounts to transparently remap file ownership for each instance.
### 9. Volume Configuration Options
Volume configuration is set per-volume and can override pool-level defaults. The most commonly used configuration keys are:
#### Size and Quotas
|
Key |
Type |
Description |
|
size |
string |
Size/quota of the volume (e.g., `10GiB`, `500MiB`). Filesystem volumes can be grown and shrunk (if enough free space). Block volumes can only be grown, never shrunk. |
#### Security and Sharing
|
Key |
Type |
Default |
Description |
|
security.shifted |
bool |
false |
Enable ID shifting overlay. Allows the volume to be attached by multiple isolated instances that use different UID/GID maps. Uses the kernel's VFS idmapped mounts to transparently remap file ownership. Essential for sharing filesystem volumes between isolated containers. |
|
security.unmapped |
bool |
false |
Disable ID mapping entirely on the volume. Flushes the current map and prevents further idmap tracking. Useful when you need to manually manage file ownership and share data between containers with incompatible maps. |
|
security.shared |
bool |
false |
Explicitly enable sharing a custom block volume across multiple instances. Use with extreme caution — only safe with cluster-aware filesystems like GFS2 or OCFS2. |
#### Snapshot Configuration
|
Key |
Type |
Default |
Description |
|
snapshots.schedule |
string |
(disabled) |
Cron expression or schedule alias for automatic snapshots. Examples:
- `0 6 * * *` — Daily at 06:00
- `0 */4 * * *` — Every 4 hours
- `@daily`, `@hourly`, `@weekly`, `@monthly` — Schedule aliases
|
|
snapshots.expiry |
string |
(keep forever) |
Auto-deletion timing for scheduled snapshots. Expression format: `1M` (minutes), `2H` (hours), `3d` (days), `4w` (weeks), `5m` (months), `6y` (years). Example: `2w 3d` means 2 weeks and 3 days. |
|
snapshots.expiry.manual |
string |
(keep forever) |
Auto-deletion timing specifically for manually created snapshots. Same format as `snapshots.expiry`. |
|
snapshots.pattern |
string |
snap%d |
Naming pattern for automatic snapshots. `%d` is an auto-incrementing number (snap0, snap1, ...). Supports Pongo2 templates, e.g., `{{ creation_date|date:'2006-01-02_15-04-05' }}` for timestamp-based names. |
#### Initial Volume Ownership (Filesystem Volumes)
|
Key |
Type |
Default |
Description |
|
initial.uid |
int |
0 |
UID of the volume owner as seen inside the instance. |
|
initial.gid |
int |
0 |
GID of the volume owner as seen inside the instance. |
|
initial.mode |
int |
711 |
Filesystem permissions (mode) of the volume root directory inside the instance. |
#### Block Device Options
|
Key |
Type |
Default |
Description |
|
block.filesystem |
string |
ext4 |
Filesystem type for block-backed filesystem volumes. Options: `btrfs`, `ext4`, `xfs`. |
|
block.mount_options |
string |
(driver default) |
Mount options for block-backed filesystem volumes. |
### 10. Snapshots
Snapshots capture the state of a volume at a point in time. They are useful for backups, testing, and rollback. Snapshot behaviour varies by storage driver.
#### Creating Snapshots
Snapshots can be created manually via the CLI:
```
`incus storage volume snapshot create <pool> <volume> [<snapshot_name>]`
```
If no name is specified, the naming pattern from `snapshots.pattern` is used (default: `snap0`, `snap1`, etc.).
#### Automatic Snapshots
Configure snapshots.schedule on the volume to enable automatic snapshots at regular intervals. Combine with snapshots.expiry to automatically delete old snapshots and prevent unbounded growth.
Example configuration for daily snapshots kept for 7 days:
```
`snapshots.schedule = @daily
snapshots.expiry = 7d
snapshots.pattern = {{ creation_date|date:'2006-01-02_15-04-05' }}`
```
#### Restoring from a Snapshot
To restore a volume to a previous snapshot state:
```
`incus storage volume snapshot restore <pool> <volume> <snapshot_name>`
```
**Important:** All instances using the volume must be **stopped** before restoring a snapshot.
#### Restoring to a New Volume
Instead of overwriting the current volume, you can restore a snapshot into a brand new volume:
```
`incus storage volume copy <pool>/<volume>/<snapshot> <target_pool>/<new_volume>`
```
This is useful for testing or when you need to compare the snapshotted state with the current state.
#### Snapshot Efficiency by Driver
|
Driver |
Snapshot Speed |
Space Efficiency |
Notes |
|
**ZFS** |
Instant |
Excellent (copy-on-write) |
Can only restore from the *latest* snapshot. To restore from older snapshots, either remove newer ones first, or copy the snapshot into a new volume. |
|
**Btrfs** |
Instant |
Excellent (copy-on-write) |
Can restore from any snapshot. |
|
**LVM (thin)** |
Fast |
Good (thin provisioning) |
Requires thin pool mode. Can restore from any snapshot. |
|
**Ceph** |
Fast |
Good (copy-on-write) |
Can restore from any snapshot. |
|
**dir** |
Slow (full copy) |
Poor (full copy) |
Every snapshot is a complete copy. Can restore from any snapshot. |
**Reliability warning:** Snapshots are stored in the **same storage pool** as the volume. If the pool fails (e.g., disk failure without RAID), both the volume and its snapshots are lost. For disaster recovery, also use **export files** (see Exporting and Importing Volumes) or copy volumes to a remote server.
### 11. Volumes in a Cluster
How volumes behave in a cluster depends on the storage driver:
#### Local Storage Drivers (dir, btrfs, lvm, zfs)
- Volumes exist **only on the specific cluster member** where they were created.
- The **Location** column in the table shows which member hosts the volume.
- When creating a volume, you may need to specify a target member. Without targeting, the volume is created on the member handling the API request.
- Multiple members can have volumes with the same name (common for image cache volumes).
- Instances using local volumes **cannot be live-migrated** — the volume does not exist on other members.
#### Shared/Distributed Storage Drivers (ceph, cephfs, lvmcluster, linstor, truenas)
- Volumes are **accessible from any cluster member**.
- The Location column will typically be empty or show a shared indicator.
- No targeting is needed when creating or accessing volumes.
- Enables **live migration** of instances between cluster members (since the volume is available everywhere).
**Ceph-specific note:** With Ceph RBD (`ceph` driver), custom filesystem volumes simulate a filesystem on top of a block device. Because of this, they can only be attached to **one instance at a time**. If you need truly shared filesystem volumes with Ceph, use CephFS (`cephfs` driver) instead.
### 12. Moving and Copying Volumes
#### Copy a Volume Between Pools
Copy a custom volume from one pool to another (or within the same pool with a different name):
```
`incus storage volume copy <source_pool>/<volume> <target_pool>/<new_name>`
```
Flags:
- `--volume-only` — Copy only the volume, skip snapshots
- `--refresh` — Update an existing copy at the destination (incremental)
**Note:** When copying between pools with different storage drivers (e.g., ZFS to LVM), the volume is **automatically converted**. No special flags needed.
#### Move or Rename a Volume
```
`incus storage volume move <source_pool>/<volume> <target_pool>/<new_name>`
```
- Same pool with different name = **rename**
- Different pools = **move** (can also rename simultaneously)
**Important:** All instances using the volume must be **stopped** before moving or renaming.
#### Between Cluster Members
For local storage drivers, you can copy or move volumes between cluster members:
```
`incus storage volume copy <pool>/<vol> <pool>/<vol> --target=<source_member> --destination-target=<target_member>`
```
This does not apply to shared drivers (Ceph, CephFS, etc.) where volumes are already accessible from all members.
#### Between Incus Servers
Copy or move volumes between completely separate Incus servers:
```
`incus storage volume copy <source_remote>:<pool>/<vol> <target_remote>:<pool>/<vol>`
```
Transfer modes via `--mode`: `pull` (default), `push`, or `relay`.
#### Move an Instance's Root Volume to Another Pool
To move an instance (and its root volume) to a different storage pool:
```
`incus move <instance_name> --storage <target_pool>`
```
The instance must be stopped first.
### 13. Exporting and Importing Volumes
For offline backup or transfer between environments, you can export volumes to archive files and import them back.
#### Export a Volume
```
`incus storage volume export <pool> <volume> [<output_file>]`
```
Default output: `backup.tar.gz` in the current directory.
Flags:
- `--compression` — Change algorithm (default: gzip). Options include `bzip2`, `none`
- `--optimized-storage` — For Btrfs/ZFS, create a driver-specific binary format (faster export and import, but only usable with the same driver type)
- `--volume-only` — Exclude snapshots from the export
#### Import a Volume
```
`incus storage volume import <pool> <file_path> [<volume_name>]`
```
If no name is specified, the original name from the export is used.
#### Backup Strategy Comparison
|
Method |
Speed |
Reliability |
Cross-Pool |
Off-Site |
|
**Snapshots** |
Fast (copy-on-write drivers) |
Low — same pool as data |
No |
No |
|
**Export files** |
Slower (full export) |
High — independent file |
Yes |
Yes (copy file off-site) |
|
**Copy to remote server** |
Varies |
High — separate server |
Yes |
Yes |
**Important:** Custom volumes attached to instances are **not** included in instance backups. They must be backed up separately using volume export or volume copy commands.
### 14. Best Practices
#### Separate Application Data from Root Disks
- Use custom volumes for databases, user uploads, application logs, and any data that should survive instance recreation.
- When you need to rebuild or replace an instance, the custom volume is unaffected — simply attach it to the new instance.
#### Set Size Limits
- Always set explicit size values on custom volumes in production. This prevents a single volume from consuming the entire pool.
- Filesystem volumes can be resized later (grown or shrunk). Block volumes can only be grown.
#### Configure Snapshot Policies
- Set snapshots.schedule for automatic snapshots at regular intervals.
- Always pair with snapshots.expiry to automatically rotate old snapshots. Without expiry, snapshots accumulate indefinitely and consume pool space.
- Use timestamp-based naming patterns for easier identification of when a snapshot was taken.
#### Plan Your Backup Strategy
- Snapshots alone are not a backup — they live in the same pool as the data.
- Periodically export critical volumes with `incus storage volume export` and store the archives on separate storage.
- For critical data, copy volumes to a remote Incus server for geographic redundancy.
- Remember that custom volumes are **not** included in instance backups — back them up separately.
#### Use `security.shifted` for Shared Volumes
- When sharing a filesystem volume between isolated containers that have different UID/GID maps, enable security.shifted on the volume.
- This uses the kernel's VFS idmapped mounts to transparently handle ownership differences.
- Without this setting, files created by one container may appear as owned by the wrong user in another container.
#### Audit Unused Volumes
- Periodically check for custom volumes with **Used By: 0**. These may be leftovers from deleted instances that are consuming pool space.
- Use the Type filter set to **Custom** and look for volumes with a Used By count of 0.
#### Choose the Right Content Type
- Use **filesystem** for most purposes — it is the most flexible and supports sharing.
- Use **block** only when the application inside the VM specifically requires raw block device access.
- Remember that the content type **cannot be changed** after creation.
#### In Clusters, Plan Volume Placement
- For local storage drivers, document which member hosts which volumes.
- Use shared drivers (Ceph, CephFS, LINSTOR) for volumes that need to be accessible from any member or that support live-migratable instances.
- When using Ceph RBD for shared filesystem volumes, be aware that each volume can only be attached to one instance at a time. Use CephFS for truly shared filesystem volumes.
---
## Incus – Custom Isos
*Source: `/static/help_incus_custom_isos.html`*
AETHER INCUS Custom ISOs Guide
## AETHER INCUS Custom ISOs Guide
This guide explains what ISO volumes are in Incus, how to upload and manage them in AETHER, and how they are used with virtual machines.
#### Table of Contents
- 1. What Are Custom ISOs?
- 2. Common Use Cases
- 3. Table Columns Explained
- 4. Uploading an ISO
- 5. Using ISOs with Virtual Machines
- 6. Boot Priority and Installation Workflow
- 7. Windows ISO Considerations
- 8. Cluster Behavior
- 9. Deleting ISOs
- 10. Best Practices
### 1. What Are Custom ISOs?
Custom ISOs in Incus are **storage volumes with content type iso**. They allow you to upload `.iso` disc image files into an Incus storage pool so they can be attached to virtual machines, just like inserting a CD/DVD into a physical server.
Key characteristics of ISO volumes:
- **Read-only** — Once uploaded, the ISO content cannot be modified. This ensures the integrity of installation media.
- **Virtual machine only** — ISOs can only be attached to VMs (not containers), as containers do not emulate optical drives.
- **Shareable** — A single ISO volume can be attached to multiple VMs simultaneously, since it is read-only.
- **Stored in pools** — ISOs are stored as volumes within your existing storage pools, using the same storage backend as your other data.
**ISO vs. Image:** Do not confuse Custom ISOs with Incus *Images* (shown in the Images tab). Images are pre-built OS templates used to launch new instances. ISOs are raw disc images typically used for OS installation or providing drivers to a VM.
ISO Upload Flow:
.iso file on Upload via Stored as ISO Attach to VM
your computer --> AETHER UI --> volume in pool --> as virtual disc
|
v
Read-only volume
(content type: iso)
### 2. Common Use Cases
#### Operating System Installation
The most common use of ISOs is to install an operating system on a new virtual machine. You upload the OS installation ISO (e.g., Ubuntu Server, Debian, Rocky Linux, Windows Server) and attach it to an empty VM to boot from it and run the installer.
#### VirtIO Drivers for Windows
Windows VMs running on Incus require **VirtIO drivers** to communicate with the virtualized hardware (disk, network, display). The VirtIO driver ISO can be attached as a second disc alongside the Windows installation ISO, allowing you to load the drivers during the Windows setup process.
**Where to get VirtIO drivers:** The official VirtIO driver ISO is available from the Fedora virtio-win project. Look for the latest `virtio-win.iso` stable release.
#### Rescue and Recovery Media
If a VM's operating system becomes unbootable, you can attach a rescue ISO (e.g., SystemRescue, GParted Live) to boot from and repair the system, resize partitions, or recover data.
#### Utility Discs
ISOs can also be used for utility purposes such as network diagnostic tools, firmware updates, or any software that comes in ISO format.
### 3. Table Columns Explained
|
Column |
Description |
|
**Name** |
The name of the ISO volume. By default, this is the original filename (without the `.iso` extension). You can override this with a custom name during upload. |
|
**Pool** |
The storage pool where this ISO volume is stored. This is the pool you selected when uploading the ISO. |
|
**Used By** |
The number of instances (VMs) that currently have this ISO attached. A count of **0** means the ISO is not in use. Since ISOs are read-only, the same ISO can be attached to multiple VMs simultaneously. |
|
**Location** |
In a cluster environment, this shows which cluster member the ISO is stored on. For shared storage pools (e.g., Ceph), this may show as empty or shared. For local storage, the ISO only exists on the specific member shown. |
|
**Created** |
The date and time when the ISO was uploaded to the storage pool. |
|
**Actions** |
Available operations for this ISO. Currently this includes a **Delete** action to remove the ISO from the storage pool. |
### 4. Uploading an ISO
To upload an ISO file, click the **Upload ISO** button in the tab header. This opens the upload dialog.
#### Upload Dialog Fields
|
Field |
Required |
Description |
|
**Storage Pool** |
Yes |
Select which storage pool to store the ISO in. The dropdown lists all available storage pools from the connected cluster. Choose a pool that has sufficient free space for the ISO file. |
|
**ISO Name** |
No |
An optional custom name for the ISO volume. If left blank, the original filename (without the `.iso` extension) is used. Only alphanumeric characters, hyphens, and underscores are recommended. |
|
**ISO File** |
Yes |
Click **Choose File** to select a `.iso` file from your computer. Only files with the `.iso` extension are accepted. |
#### Upload Process
- Click **Upload ISO** in the tab header.
- Select a **Storage Pool** from the dropdown.
- Optionally enter a custom **ISO Name**.
- Click **Choose File** and select your `.iso` file.
- Click **Upload**. A progress bar will show the upload progress.
- Wait for the upload to complete. The ISO will appear in the table once finished.
**Upload time:** ISO files can be large (typically 500 MB to 6 GB). Upload time depends on your network connection speed to the AETHER server. The progress bar tracks the actual upload to the server, which then transfers the data to the Incus storage pool.
**Duplicate names:** Each ISO volume name must be unique within a storage pool. If you try to upload an ISO with a name that already exists in the selected pool, the upload will fail. Use a different name or delete the existing ISO first.
### 5. Using ISOs with Virtual Machines
After uploading an ISO, you can attach it to a virtual machine. The ISO appears to the VM as a virtual CD/DVD drive.
#### Attaching an ISO to a VM
ISO attachment is done from the **Instances** tab. When editing or creating a VM, you can add a disk device that references the ISO volume. The configuration looks like:
|
Setting |
Value |
|
pool |
The storage pool where the ISO is stored |
|
source |
The name of the ISO volume |
|
type |
`disk` |
Incus CLI equivalent:
```
`incus config device add my-vm install-media disk pool=default source=ubuntu-24.04 io.bus=virtio`
```
#### Key Behaviors
- **Multiple ISOs:** You can attach more than one ISO to a single VM (e.g., OS installer + VirtIO drivers).
- **Shared access:** The same ISO can be attached to many VMs simultaneously since it is read-only.
- **Hot-attach:** ISOs can be attached and detached while the VM is running.
- **Detaching:** After installation is complete, remove the ISO device from the VM so it no longer boots from the disc.
### 6. Boot Priority and Installation Workflow
When a VM has an ISO attached, it affects the boot order:
- **First boot (empty disk):** The VM will automatically boot from the ISO since the root disk has no bootable OS yet.
- **Installation:** Run through the OS installer as you normally would. The installer writes to the VM's root disk.
- **Post-installation:** After installation, the VM will typically boot from the root disk. However, it is good practice to detach the ISO to avoid accidentally booting into the installer again.
**UEFI boot order:** Incus VMs use UEFI firmware. The boot order generally prefers the hard disk once an OS is installed. If the VM keeps booting from the ISO instead of the installed OS, detach the ISO device.
#### Typical Installation Workflow
1. Upload ISO Upload your OS installer to a storage pool
|
2. Create VM Create a new VM with a root disk
|
3. Attach ISO Add the ISO as a disk device to the VM
|
4. Start VM VM boots from the ISO installer
|
5. Install OS Complete the OS installation to the root disk
|
6. Reboot VM reboots into the installed OS
|
7. Detach ISO Remove the ISO device from the VM
### 7. Windows ISO Considerations
Installing Windows on Incus VMs requires additional preparation compared to Linux installations.
#### VirtIO Drivers
Windows does not include drivers for VirtIO hardware by default. During the Windows installation, the installer will not see the VM's disk or network unless VirtIO drivers are loaded. To solve this:
- Upload the **Windows installation ISO** to your storage pool.
- Upload the **VirtIO driver ISO** (`virtio-win.iso`) to the same or another pool.
- Attach **both ISOs** to the VM as separate disk devices.
- Start the VM and begin Windows Setup.
- When the installer asks "Where do you want to install Windows?" and shows no drives, click **Load driver**.
- Browse to the VirtIO disc, navigate to the appropriate folder (e.g., `viostor\w11\amd64`), and load the storage driver.
- Optionally also load the network driver (`NetKVM`) for network access during setup.
- Complete the installation as normal.
**After installation:** Once Windows is installed and running, install the full VirtIO guest tools package from inside the VM. This provides optimized drivers for display (QXL/VirtIO-GPU), network, balloon memory, and serial console. The installer is available on the VirtIO ISO under the `guest-agent` or `virtio-win-gt` folder.
#### Windows ISO Repacking
For advanced users, it is possible to repack a Windows ISO to include VirtIO drivers directly in the installation image. This eliminates the need for a second ISO during setup. Tools like `oscdimg` (from the Windows ADK) or `mkisofs` can be used to rebuild the ISO. This is useful when deploying many Windows VMs.
### 8. Cluster Behavior
How ISOs behave in a cluster depends on the type of storage pool used:
#### Shared Storage (e.g., Ceph, CephFS)
- The ISO is available on **all cluster members** automatically.
- Any VM on any member can attach and use the ISO.
- The **Location** column may show as empty or indicate shared availability.
#### Local Storage (e.g., dir, zfs, lvm, btrfs)
- The ISO only exists on the **specific cluster member** where it was uploaded.
- Only VMs running on that same member can attach the ISO.
- The **Location** column shows which member the ISO is stored on.
- To use the ISO on another member, you would need to upload it again to that member's local pool.
**Recommendation:** If you are running a cluster and need ISOs accessible from all members, use a shared storage pool (e.g., Ceph) for your ISO uploads. This avoids having to upload the same ISO to each member individually.
### 9. Deleting ISOs
To delete an ISO, click the **Delete** button in the Actions column for that ISO.
#### Deletion Rules
- **Used By must be 0:** You cannot delete an ISO that is currently attached to one or more VMs. First detach the ISO from all VMs, then delete it.
- **Permanent:** Deleting an ISO removes it from the storage pool entirely. There is no undo — you would need to re-upload the ISO file.
- **Confirmation:** AETHER will ask for confirmation before deleting an ISO.
**Caution:** Make sure no VMs depend on the ISO before deleting. If a VM is configured to boot from an ISO that has been deleted, it may fail to start.
### 10. Best Practices
|
Practice |
Recommendation |
|
**Naming** |
Use descriptive names that include the OS and version, e.g., `ubuntu-24.04-server`, `win2022-eval`, `virtio-win-0.1.262`. This makes it easy to identify ISOs at a glance. |
|
**Cleanup** |
Remove ISOs that are no longer needed. They consume storage pool space. Check the **Used By** column — ISOs with a count of 0 that are outdated can safely be deleted. |
|
**Version tracking** |
Include version numbers in ISO names. When a new version is released, upload it alongside the old one, migrate VMs if needed, then delete the old ISO. |
|
**Shared storage for clusters** |
In cluster environments, upload ISOs to shared storage pools so all members can access them without duplicating the ISO on each member. |
|
**Keep essentials available** |
Always keep a rescue ISO and VirtIO driver ISO uploaded. These are invaluable when troubleshooting VM issues or setting up Windows VMs. |
|
**Detach after install** |
Always detach installation ISOs from VMs after the OS installation is complete. Leaving them attached wastes a device slot and may cause unintended boot behavior. |
---
## Incus – Storage Buckets
*Source: `/static/help_incus_storage_buckets.html`*
AETHER INCUS Storage Buckets Guide
## AETHER INCUS Storage Buckets Guide
This guide explains what storage buckets are in Incus, how they provide S3-compatible object storage, and how to create and manage buckets and access keys in AETHER Infrastructure Management.
#### Table of Contents
- 1. What Are Storage Buckets?
- 2. How Buckets Differ from Volumes
- 3. Supported Storage Drivers
- 4. Table Columns Explained
- 5. Creating a Bucket
- 6. S3 Endpoint Configuration
- 7. Managing Bucket Keys
- 8. Connecting with S3 Clients
- 9. Common Use Cases
- 10. Cluster Behavior
- 11. Deleting Buckets
- 12. Limitations & Caveats
- 13. Best Practices
### 1. What Are Storage Buckets?
Storage buckets in Incus provide **S3-compatible object storage** directly within your Incus infrastructure. They allow applications to store and retrieve unstructured data (files, images, logs, backups, etc.) using the standard S3 protocol — the same protocol used by Amazon S3, MinIO, and most cloud storage services.
Key characteristics of storage buckets:
- **Not attached to instances** — Unlike storage volumes that are mounted to containers or VMs, buckets are accessed over the network via a URL.
- **S3 API access** — Applications connect using standard S3 tools and libraries with access keys and secret keys.
- **Stored in pools** — Buckets live inside your existing storage pools, using the same storage backend as your other data.
- **Key-based access control** — Each bucket has one or more access key pairs with defined roles (admin or read-only).
How Buckets Work:
Application/Tool S3 Protocol (HTTPS) Incus Storage Pool
(aws cli, mc, ----> Access Key + Secret ----> Bucket
rclone, s3cmd, Key authentication (objects/files stored
any S3 library) within the pool)
**Buckets vs. Volumes vs. ISOs:** Incus storage pools can hold three types of content: *volumes* (mounted to instances), *ISOs* (attached as virtual discs to VMs), and *buckets* (accessed over the network via S3). Each serves a different access pattern.
### 2. How Buckets Differ from Volumes
|
Feature |
Storage Volume |
Storage Bucket |
|
**Access method** |
Mounted as a filesystem inside an instance |
Accessed over the network via S3 API (HTTPS) |
|
**Attached to instance?** |
Yes — must be attached to a container or VM |
No — independent, accessed by URL |
|
**Authentication** |
Instance access controls (no extra credentials) |
Requires S3 access key + secret key |
|
**Multi-client access** |
Typically one instance at a time (unless shared) |
Multiple applications can access simultaneously |
|
**Data model** |
Filesystem (directories, files, permissions) |
Object storage (flat namespace, key-value) |
|
**Best for** |
Databases, application data, home directories |
Backups, media, logs, shared data, cloud-native apps |
### 3. Supported Storage Drivers
Not all storage pool drivers support buckets. The following table shows which drivers can be used:
|
Driver |
Bucket Support |
Quota (`size`) Support |
Notes |
|
**dir** (Directory) |
Yes |
No |
Simplest driver. No quota enforcement for buckets. |
|
**btrfs** |
Yes |
Yes |
Good local option with quota support. |
|
**lvm** |
Yes |
Yes |
Supports quotas via logical volume sizing. |
|
**zfs** |
Yes |
Yes |
Most reliable local option with native quota support. |
|
**cephobject** |
Yes |
Yes |
Bucket-only driver using Ceph RADOS Gateway. Cannot store volumes. |
|
**ceph** (RBD) |
No |
— |
Block storage only, no bucket support. |
|
**cephfs** |
No |
— |
Filesystem storage only, no bucket support. |
#### How Buckets Are Served
Incus uses two different S3 backends depending on the pool type:
- **Local drivers** (dir, btrfs, lvm, zfs): Incus uses an **embedded MinIO process** per bucket to serve the S3 API. MinIO processes are launched on-demand and stopped after an idle period to conserve memory.
- **cephobject**: Uses the **Ceph RADOS Gateway (radosgw)** as the S3 interface. Requires a pre-existing, externally configured Ceph cluster with radosgw running.
### 4. Table Columns Explained
|
Column |
Description |
|
**Name** |
The name of the storage bucket. Used to identify the bucket in S3 URLs and API calls. |
|
**Pool** |
The storage pool where this bucket is stored. |
|
**Description** |
An optional description of the bucket's purpose. |
|
**Size** |
The quota (maximum size) set for this bucket. If no quota is set, the bucket can grow to use all available space in the pool. The `dir` driver does not support quotas. |
|
**Keys** |
The number of access key pairs created for this bucket. Each key pair provides S3 credentials (access key + secret key) for applications to connect. |
|
**Location** |
In a cluster, shows which member the bucket is stored on. For `cephobject` pools, buckets are accessible from all members. |
|
**Actions** |
Available operations: **View/Keys** to see bucket details and manage access keys, **Delete** to remove the bucket. |
### 5. Creating a Bucket
Click the **Create Bucket** button in the tab header to open the creation dialog.
#### Create Bucket Dialog Fields
|
Field |
Required |
Description |
|
**Storage Pool** |
Yes |
Select which storage pool to create the bucket in. Only pools that support buckets will work (see Supported Storage Drivers). |
|
**Bucket Name** |
Yes |
A unique name for the bucket. Use lowercase letters, numbers, and hyphens. The name must be unique within the storage pool. |
|
**Description** |
No |
An optional description to help identify the bucket's purpose. |
|
**Size Quota** |
No |
Maximum size for the bucket (e.g., `10GiB`). Leave empty for no quota (unlimited within pool capacity). Not supported on the `dir` driver. |
#### Creation Process
- Click **Create Bucket** in the tab header.
- Select a **Storage Pool** from the dropdown.
- Enter a **Bucket Name**.
- Optionally add a **Description** and **Size Quota**.
- Click **Create**.
- When the bucket is created, an initial **admin key** is automatically generated. A dialog will show you the **access key** and **secret key**. Save these credentials immediately.
**Save your credentials!** The secret key is only shown once at creation time. If you lose it, you will need to create a new key. Copy both the access key and secret key and store them securely.
### 6. S3 Endpoint Configuration
Before applications can connect to buckets on **local storage pools** (dir, btrfs, lvm, zfs), the Incus server must be configured with an S3 listening address.
#### Setting the S3 Address
This is a server-level configuration done via CLI:
```
`incus config set core.storage_buckets_address :8555`
```
This tells Incus to listen on port 8555 for incoming S3 requests. You can use any available port.
**Required for local drivers:** If core.storage_buckets_address is not set, local buckets will not be accessible via S3. This setting is not needed for `cephobject` pools, which use the Ceph RADOS Gateway endpoint directly.
#### The S3 URL
Once configured, the S3 endpoint URL for your buckets will be:
```
`https://<server-ip-or-hostname>:8555`
```
Incus enforces HTTPS on all endpoints. The S3 endpoint uses the same TLS certificate as the main Incus API. If using self-signed certificates, you may need to configure your S3 client to skip certificate verification.
The S3 URL for each bucket is also displayed in the **View/Keys** dialog in AETHER.
### 7. Managing Bucket Keys
Applications access buckets using **S3 credentials**: an access key (public identifier) and a secret key (private secret). You can create multiple key pairs per bucket, allowing different applications to have different access levels.
#### Viewing and Managing Keys
Click the **View/Keys** button in the Actions column to open the bucket detail view. This shows:
- Bucket properties (name, pool, description, location, S3 URL)
- A table of all access keys with their names, roles, and access key IDs
- An **Add Key** button to create new keys
#### Key Roles
Each key is assigned one of two roles:
|
Role |
Permissions |
Use Case |
|
Admin |
Full access: list, get, put, and delete objects |
Applications that need to write data to the bucket |
|
Read Only |
Read-only access: list and get objects only |
Applications that only need to read/download data |
#### Creating a New Key
- Open the bucket via **View/Keys** in the Actions column.
- Click **Add Key**.
- Enter a **Key Name** (a label for your reference, not sent to S3 clients).
- Optionally add a **Description**.
- Select the **Role** (Admin or Read Only).
- Click **Create Key**.
- A dialog will show the generated **access key** and **secret key**. Copy and save both immediately.
**Secret keys are shown only once!** After you close the credentials dialog, the secret key cannot be retrieved again. If you lose a secret key, delete the key and create a new one.
**Multiple keys per bucket:** You can create as many key pairs as you need. This allows you to give each application its own credentials, making it easy to revoke access for a single application without affecting others.
#### Deleting a Key
Click the **Delete** button next to a key in the keys table. This immediately revokes access for any application using that key pair. The bucket and its data remain unaffected.
### 8. Connecting with S3 Clients
After creating a bucket and obtaining your access key and secret key, you can connect using any S3-compatible tool. Incus buckets use **path-style addressing** (not virtual-hosted-style), so ensure your client is configured accordingly.
#### MinIO Client (mc)
```
`# Set up a connection alias
mc alias set incus https://<server>:8555 <access_key> <secret_key> --api S3v4 --insecure
# List objects in bucket
mc ls incus/<bucket_name>
# Upload a file
mc cp myfile.txt incus/<bucket_name>/
# Download a file
mc cp incus/<bucket_name>/myfile.txt ./
# Mirror a directory
mc mirror ./localdir incus/<bucket_name>/`
```
#### AWS CLI
```
`# Configure credentials (enter access key, secret key, region: us-east-1)
aws configure
# List objects in bucket
aws --endpoint-url https://<server>:8555 s3 ls s3://<bucket_name> --no-verify-ssl
# Upload a file
aws --endpoint-url https://<server>:8555 s3 cp myfile.txt s3://<bucket_name>/ --no-verify-ssl
# Download a file
aws --endpoint-url https://<server>:8555 s3 cp s3://<bucket_name>/myfile.txt ./ --no-verify-ssl
# Sync a directory
aws --endpoint-url https://<server>:8555 s3 sync ./localdir s3://<bucket_name>/ --no-verify-ssl`
```
#### s3cmd
```
`# Create ~/.s3cfg with:
[default]
access_key = <access_key>
secret_key = <secret_key>
host_base = <server>:8555
host_bucket = <server>:8555
use_https = True
check_ssl_certificate = False
check_ssl_hostname = False
signature_v2 = False
# Then use:
s3cmd ls s3://<bucket_name>
s3cmd put myfile.txt s3://<bucket_name>/
s3cmd get s3://<bucket_name>/myfile.txt`
```
#### rclone
```
`# Add to ~/.config/rclone/rclone.conf:
[incus]
type = s3
provider = Other
env_auth = false
access_key_id = <access_key>
secret_access_key = <secret_key>
endpoint = https://<server>:8555
force_path_style = true
acl = private
# Then use:
rclone ls incus:<bucket_name>
rclone copy myfile.txt incus:<bucket_name>/
rclone sync ./localdir incus:<bucket_name>/`
```
**Self-signed certificates:** Incus typically uses self-signed TLS certificates. The `--insecure` or `--no-verify-ssl` flags in the examples above disable certificate verification. For production environments, consider using valid certificates (e.g., via Let's Encrypt or `incus cluster update-certificate`).
**From inside an Incus instance:** Instances can reach the S3 endpoint through the gateway address. However, the default gateway hostname (`_gateway.incus`) contains an underscore which some S3 libraries may reject as an invalid hostname. A workaround is to add a `/etc/hosts` entry mapping the gateway IP to a valid hostname.
### 9. Common Use Cases
##### Application Data Sharing
Multiple instances or external applications need shared access to the same data store. Each application gets its own S3 credentials with the appropriate role, enabling controlled concurrent access without mounting shared volumes.
##### Cloud-Native Application Support
Many modern applications (web apps, microservices, Kubernetes workloads) are designed to work with S3-compatible storage. Incus buckets let you provide this locally without depending on an external cloud service.
##### Backup Destination
Applications can push backups to a bucket using standard S3 tools like rclone or restic. This provides a separate storage location for backups that is independent of instance volumes.
##### Media and Asset Storage
Storing images, videos, documents, or other unstructured data that needs to be accessed by multiple services or users over the network.
##### Log Aggregation
Centralized log storage where multiple instances push logs to a shared bucket, accessible by monitoring and analysis tools.
##### Local S3 for Development and Testing
Providing S3-compatible storage locally for development and testing workflows, eliminating the need for AWS or another cloud provider during development.
### 10. Cluster Behavior
How buckets behave in a cluster depends on the storage pool driver:
#### Local Storage Drivers (dir, btrfs, lvm, zfs)
- A bucket exists **only on the cluster member** where it was created.
- Applications must connect to that specific member's S3 endpoint to access the bucket.
- The **Location** column shows which member the bucket is stored on.
- To have the same bucket available on multiple members, you must create separate buckets on each member.
#### cephobject Driver
- Buckets are **available from any cluster member** because Ceph provides distributed storage.
- This is the main advantage of cephobject in a cluster — all members share the same bucket data.
- Applications can connect to any member's S3 endpoint.
**Cluster recommendation:** If you need buckets accessible from all cluster members, use a `cephobject` storage pool. For local pools, each bucket only exists on one member.
### 11. Deleting Buckets
Click the **Delete** button in the Actions column for a bucket.
**Deletion is permanent and destroys all data!** Deleting a bucket removes:
- All objects (files) stored in the bucket
- All access keys associated with the bucket
There is no undo. Make sure you have backups of any data you need before deleting a bucket.
AETHER will ask for confirmation before proceeding. The confirmation dialog warns that all access keys will also be deleted.
### 12. Limitations & Caveats
|
Limitation |
Details |
|
**dir driver has no quotas** |
The `dir` storage driver does not support the size setting for buckets. Buckets on `dir` pools can grow without limit until the pool's underlying filesystem is full. |
|
**Memory overhead (local drivers)** |
Each active bucket on a local storage pool launches an embedded MinIO process that uses approximately **~100 MB of memory**. MinIO processes are started on-demand and stopped after an idle period, but many simultaneously active buckets can consume significant memory. |
|
**No bucket creation via S3 API** |
Buckets can only be created through the Incus API or CLI (or AETHER UI). The S3 endpoint only allows operations on *objects within* existing buckets — it cannot create new buckets. |
|
**Limited access control** |
Only two key roles are available: *admin* (full access) and *read-only* (list and get only). There are no granular per-object or per-prefix policies through Incus. |
|
**Cannot shrink below used size** |
If you set a quota on a bucket, you cannot reduce it below the bucket's current used size. You must delete objects first to free space. |
|
**HTTPS enforced** |
Incus does not support cleartext (HTTP) S3 endpoints. All connections must use HTTPS. Self-signed TLS certificates may require extra client configuration. |
|
**DNS underscore issue** |
When accessing buckets from inside Incus instances, the default gateway hostname (`_gateway.incus`) contains an underscore, which some S3 libraries reject as an invalid hostname. |
|
**Path-style addressing only** |
Incus local buckets use path-style S3 addressing (e.g., `https://server:8555/bucket-name/object`), not virtual-hosted-style. Ensure your S3 client is configured for path-style. |
### 13. Best Practices
|
Practice |
Recommendation |
|
**Naming** |
Use lowercase, descriptive names with hyphens (e.g., `app-backups`, `media-assets`, `logs-production`). S3 bucket names are typically case-sensitive and should avoid special characters. |
|
**Set quotas** |
Always set a size quota on buckets (except on `dir` pools which don't support it). This prevents a single bucket from consuming all pool space and affecting other volumes and instances. |
|
**Least privilege keys** |
Create *read-only* keys for applications that only need to read data. Only use *admin* keys for applications that must write. This limits the impact if a key is compromised. |
|
**One key per application** |
Give each application its own key pair rather than sharing credentials. This makes it easy to revoke access for a single application by deleting its key without affecting others. |
|
**Key rotation** |
Periodically create new keys and update applications, then delete old keys. This limits the exposure window if a key is leaked. |
|
**Secure credential storage** |
Never store S3 credentials in source code or unencrypted configuration files. Use environment variables, secrets managers, or encrypted config files. |
|
**Shared storage for clusters** |
In cluster environments, use `cephobject` pools for buckets that need to be accessible from all members. Local pool buckets only exist on one member. |
|
**Monitor bucket sizes** |
Regularly check bucket usage against quotas. Buckets without quotas can grow unexpectedly, especially when used for logs or backups. |
---
## Incus – Profiles
*Source: `/static/help_incus_profiles.html`*
AETHER INCUS Profiles Guide
## AETHER INCUS Profiles Guide
This guide explains what Incus profiles are, how they work as reusable configuration templates, and how to create and manage them in AETHER Infrastructure Management.
#### Table of Contents
- 1. What Are Profiles?
- 2. The Default Profile
- 3. How Profile Stacking Works
- 4. Table Columns Explained
- 5. Creating a Profile
- 6. Profile Configuration Tabs
- 6.1 Main Configuration
- 6.2 Disk Devices
- 6.3 Network Devices
- 6.4 Resource Limits
- 6.5 Security Policies
- 6.6 Snapshots
- 6.7 Migration
- 6.8 Boot
- 6.9 Cloud Init
- 6.10 Advanced Config
- 7. Editing a Profile
- 8. Deleting a Profile
- 9. Common Profile Patterns
- 10. Profiles vs Instance Config
- 11. Cluster Behavior
- 12. Best Practices
### 1. What Are Profiles?
Profiles are **reusable configuration templates** that store a set of configuration options and device definitions. Instead of configuring the same settings on every instance individually, you define them once in a profile and apply that profile to any number of instances.
A profile contains two main sections:
- **Config** — Instance options such as resource limits, security settings, boot behavior, snapshot schedules, and cloud-init data.
- **Devices** — Hardware definitions such as network interfaces, disk mounts, GPUs, and USB devices.
Key characteristics:
- **Reusable** — A single profile can be applied to many instances.
- **Stackable** — Multiple profiles can be applied to the same instance, and they are merged in order.
- **Live propagation** — When you edit a profile, the changes are **automatically applied to all instances** using that profile.
- **Type-agnostic** — Profiles can contain options for both containers and VMs. Options not applicable to the instance type are silently ignored.
How Profiles Work:
Profile: "web-server" Applied to instances:
┌─────────────────────┐
│ Config: │ ──────> web-01 (container)
│ limits.cpu: 4 │ ──────> web-02 (container)
│ limits.memory: 4G │ ──────> web-03 (VM)
│ Devices: │
│ eth0: bridged NIC │ Edit the profile once
│ root: 50GiB disk │ = all 3 instances updated
└─────────────────────┘
**Be careful editing profiles in production.** Changes to a profile are immediately applied to ALL instances using it. Always test profile changes on a development profile or instance first.
### 2. The Default Profile
Every Incus installation has a **default** profile that provides the minimum configuration every instance needs:
- **A root disk** — Provides storage from the default pool, mounted at `/`.
- **A network interface** — Typically `eth0`, connected to the default bridge network (`incusbr0`).
The default profile has special rules:
- It is **automatically applied** when an instance is created without specifying any profiles.
- It **cannot be renamed** or **deleted**.
- It **can be modified** — you can change its config and devices to suit your environment.
**Why the default profile matters:** Without a root disk and network interface, an instance would have no storage and no network connectivity. The default profile ensures every instance has these essentials, even if no other profile is specified.
### 3. How Profile Stacking Works
Instances can have **multiple profiles** applied simultaneously. The profiles are applied in order, and when the same setting appears in more than one profile, the **last one wins**.
#### Precedence Hierarchy (Lowest to Highest)
1. First profile listed (lowest priority)
2. Second profile listed
3. ...
4. Last profile listed
5. Instance-specific config (ALWAYS wins)
#### Config Keys
For configuration keys (like limits.cpu), the last profile to set a specific key takes precedence. Instance-specific config always overrides all profiles.
**Example:** Profile A sets `limits.cpu=2`, Profile B sets `limits.cpu=4`. If an instance uses both profiles in order [A, B], the instance gets `limits.cpu=4` (Profile B wins).
#### Devices
Devices are identified by their **unique name**. If the same device name appears in a later profile, the **entire device entry is replaced** (not merged field-by-field).
**Example:** Profile A defines `eth0` as a bridged NIC, Profile B also defines `eth0` as a macvlan NIC. The instance gets Profile B's definition entirely.
#### Blocking Inherited Devices
To prevent an instance from inheriting a device from a profile without providing a replacement, you can use a device of type `none` with the same name (via CLI):
```
`incus config device add myinstance eth0 none`
```
The `none` type creates nothing inside the instance — it purely blocks inheritance of that device name.
#### Viewing Effective Configuration
To see the final computed configuration after all profile merging and overrides, use the CLI:
```
`incus config show <instance> --expanded`
```
This shows the "expanded" view with all inherited values resolved.
### 4. Table Columns Explained
|
Column |
Description |
|
**Name** |
The profile name. Must be unique. The `default` profile always exists. |
|
**Description** |
An optional text description of the profile's purpose. Shows "-" if not set. |
|
**Devices** |
A summary of devices defined in the profile, shown as badges with the device name and type (e.g., `eth0 (nic)`, `root (disk)`). |
|
**Config** |
The number of configuration keys set in the profile. Click **Edit** to see the full configuration. |
|
**Used By** |
The number of instances currently using this profile. A profile with a Used By count greater than 0 cannot be deleted. |
|
**Actions** |
Available operations: **Edit** to view and modify the profile, **Delete** to remove it (if not in use and not the default profile). |
### 5. Creating a Profile
Click the **Create Profile** button in the tab header. This opens a multi-tab dialog where you can configure all aspects of the profile.
The dialog is organized into **10 configuration tabs** accessible via a sidebar:
- **Main Configuration** — Profile name and description
- **Disk** — Root disk and additional disk devices
- **Network** — Network interface devices
- **Resource Limits** — CPU, memory, disk I/O, and process limits
- **Security Policies** — Deletion protection, privileged mode, nesting, secureboot, and more
- **Snapshots** — Automated snapshot schedules and retention
- **Migration** — Stateful migration and cluster evacuation behavior
- **Boot** — Autostart, priority, and shutdown behavior
- **Cloud Init** — Cloud-init network config, user-data, and vendor-data
- **Advanced Config** — Custom key-value pairs for any Incus config option
**You don't need to fill in every tab.** Only configure the settings relevant to your use case. Empty fields are simply not included in the profile. A profile can be as minimal or as comprehensive as needed.
### 6. Profile Configuration Tabs
##### 6.1 Main Configuration
|
Field |
Required |
Description |
|
**Profile Name** |
Yes |
A unique name for the profile. Use descriptive names like `web-server`, `dev-small`, or `snap-hourly`. Cannot be changed after creation. |
|
**Description** |
No |
A human-readable description of the profile's purpose. Helpful for team members to understand what the profile is for. |
##### 6.2 Disk Devices
Configure storage for instances using this profile.
###### Root Disk
The root disk is the primary storage for the instance, mounted at `/`. Every instance needs a root disk (usually provided by the default profile).
|
Field |
Description |
|
**Storage Pool** |
Which storage pool to use for the root disk. |
|
**Size** |
Maximum size of the root disk (e.g., 50 GiB). Leave empty for no limit (uses available pool space). |
###### Extra Disks
Click **Add Disk Device** to attach additional storage. Each extra disk needs:
- **Device Name** — A unique identifier (e.g., `data`, `backup`).
- **Storage Pool** — The pool to draw storage from.
- **Path** — Where to mount the disk inside the instance (e.g., `/data`).
- **Size** — Maximum size with unit (optional).
##### 6.3 Network Devices
Configure network interfaces for instances using this profile.
Click **Add Network Device** to add a network interface. Each device needs:
- **Device Name** — The interface name inside the instance (e.g., `eth0`, `eth1`).
- **Network** — Select from available managed networks. The network determines connectivity (bridge, OVN, etc.).
**Multiple NICs:** You can add multiple network devices to give an instance access to different networks. Each device must have a unique name.
##### 6.4 Resource Limits
Control how much CPU, memory, and other resources instances can consume.
|
Field |
Config Key |
Description |
|
**CPU Limit** |
limits.cpu |
Two modes:
- **Dynamic** — A number of virtual CPUs (e.g., `2`). The instance can use any available host CPU cores, up to this limit.
- **Fixed** — Pin to specific CPU cores (e.g., `0,1` or `0-3`). The instance can only use these exact cores. Better for performance-sensitive workloads.
|
|
**Memory Limit** |
limits.memory |
Maximum memory with unit (e.g., `4 GiB`, `512 MiB`). For containers, can also be a percentage of host memory (e.g., `50%`). |
|
**Memory Swap** |
limits.memory.swap |
Containers Whether the instance is allowed to use swap space. Set to `deny` to prevent swapping (useful for latency-sensitive workloads). |
|
**Disk Priority** |
limits.disk.priority |
I/O scheduling priority from 0 (lowest) to 10 (highest). Determines who gets disk access first when the storage is contended. |
|
**Max Processes** |
limits.processes |
Containers Maximum number of processes that can run simultaneously inside the instance. Protects against fork bombs. |
##### 6.5 Security Policies
Security settings control instance isolation, privilege levels, and protection mechanisms. Some settings apply only to containers, others only to VMs.
###### General (Both Containers and VMs)
|
Field |
Config Key |
Description |
|
**Protect Deletion** |
security.protection.delete |
Prevents the instance from being accidentally deleted. Must be disabled before the instance can be removed. Strongly recommended for production instances. |
###### Container-Specific Containers
|
Field |
Config Key |
Description |
|
**Privileged** |
security.privileged |
Run as a privileged container (UID 0 inside = UID 0 on host). **Not recommended** — significantly reduces isolation. Only use when absolutely necessary. |
|
**Nesting** |
security.nesting |
Allow running Incus or Docker inside the container. Required for Docker-in-Incus or nested virtualization use cases. |
|
**Protect UID/GID Shift** |
security.protection.shift |
Prevents the UID/GID mapping from being changed on the container's filesystem. |
|
**Base Host ID** |
security.idmap.base |
Override the base host UID/GID for the container's user namespace mapping. Usually auto-detected. |
|
**Idmap Size** |
security.idmap.size |
Size of the UID/GID range mapped into the container. Default is 65536. |
|
**Unique Idmap** |
security.idmap.isolated |
Give each container a unique UID/GID range. Improves isolation: compromising one container cannot affect another's files. |
|
**Allow /dev/lxd** |
security.devlxd |
Whether the container can access the `/dev/lxd` API socket (also named `/dev/incus` in newer versions). Used for instance-to-host communication. |
|
**/dev/lxd Images API** |
security.devlxd.images |
Whether the container can access the images API through the `/dev/lxd` socket. |
###### VM-Specific VMs
|
Field |
Config Key |
Description |
|
**Secureboot** |
security.secureboot |
Enable UEFI Secure Boot. Enabled by default. Verifies the OS bootloader is signed and trusted. Disable if the OS does not support Secure Boot. |
|
**CSM** |
security.csm |
Compatibility Support Module. Enables legacy BIOS boot mode for operating systems that do not support UEFI. Secureboot must be disabled when CSM is enabled. |
**User namespaces (idmap):** By default, Incus containers run unprivileged using user namespaces. UID 0 (root) inside the container maps to a high-numbered UID on the host (e.g., 100000). This means even if an attacker escapes the container, they have no privileges on the host. The idmap settings let you control this mapping.
##### 6.6 Snapshots
Configure automated snapshot schedules for all instances using this profile.
|
Field |
Config Key |
Description |
|
**Snapshot Name Pattern** |
snapshots.pattern |
Naming template for auto-created snapshots. Supports Pongo2 template syntax. Examples:
- `snap%d` — snap0, snap1, snap2, ...
- `auto-{{ creation_date|date:'2006-01-02' }}` — auto-2026-02-13
- `hourly-{{ creation_date|date:'15:04' }}` — hourly-14:30
|
|
**Expire After** |
snapshots.expiry |
Auto-delete snapshots after this duration. Format: combinations of `Xd` (days), `Xw` (weeks), `XH` (hours), `XM` (minutes). Examples: `7d`, `4H`, `1w 2d`. Leave empty to keep forever. |
|
**Schedule** |
snapshots.schedule |
When to take snapshots. Accepts cron expressions or aliases: `@hourly`, `@daily`, `@midnight`, `@weekly`, `@monthly`, `@annually`. Or custom cron: `0 2 * * *` (daily at 2:00 AM). |
|
**Snapshot Stopped Instances** |
snapshots.schedule.stopped |
Whether to also snapshot instances that are currently stopped. By default, only running instances are snapshotted on schedule. |
##### 6.7 Migration
Control how instances behave during migration and cluster evacuation.
|
Field |
Config Key |
Description |
|
**Stateful Migration** |
migration.stateful |
VMs Allow stateful stop/start (saves running memory state to disk). Required for live migration of VMs. Enables the instance to be paused, moved, and resumed without rebooting. |
|
**Cluster Evacuation** |
cluster.evacuate |
What happens when a cluster member is evacuated (taken offline for maintenance):
- `auto` — Incus decides the best method based on instance type and devices.
- `live-migrate` — Instance stays running during migration (minimal downtime). Requires stateful migration to be enabled for VMs.
- `migrate` — Stop the instance, move it, then start it on the new member.
- `stop` — Simply stop the instance (do not move it).
- `stateful-stop` — Save the running state to disk, stop the instance. Can be resumed later with full state.
- `force-stop` — Forcefully kill the instance immediately.
|
##### 6.8 Boot
Control instance startup and shutdown behavior.
|
Field |
Config Key |
Description |
|
**Autostart** |
boot.autostart |
Whether to start the instance when the Incus daemon starts (e.g., after a host reboot). `yes` = always start, `no` = never start, leave empty for default behavior (restore last state). |
|
**Autostart Delay** |
boot.autostart.delay |
Seconds to wait after starting this instance before starting the next one. Useful to stagger startup of dependent services (e.g., start the database before the web server). |
|
**Autostart Priority** |
boot.autostart.priority |
Startup order. Instances with **higher** priority numbers start first. Use this to ensure infrastructure services (DNS, databases) start before application instances. |
|
**Host Shutdown Timeout** |
boot.host_shutdown_timeout |
Seconds to wait for the instance to shut down gracefully when the host is shutting down. After this timeout, the instance is force-killed. Default: 30 seconds. |
|
**Stop Priority** |
boot.stop.priority |
Shutdown order. Instances with **higher** priority numbers stop first. Use this to ensure application instances stop before infrastructure services. |
**Startup ordering example:** Set your database to priority 100 with a 5-second delay, and your web server to priority 50. The database starts first, waits 5 seconds, then the web server starts.
##### 6.9 Cloud Init
Cloud-init is an industry-standard tool for automating the initial setup of cloud instances. Incus supports cloud-init for both containers and VMs (when the image includes cloud-init).
|
Field |
Config Key |
Description |
|
**Network Config** |
cloud-init.network-config |
YAML network configuration. By default, cloud-init configures DHCP on eth0. Use this to set static IPs, DNS servers, routes, VLANs, etc. |
|
**User Data** |
cloud-init.user-data |
YAML cloud-config for instance-specific provisioning. Runs commands, installs packages, creates users, writes files, etc. Typically used for per-instance customization. |
|
**Vendor Data** |
cloud-init.vendor-data |
YAML cloud-config for general/default provisioning. Same format as user-data but intended for organization-wide defaults. Best practice: put vendor-data in profiles, user-data in instance config. |
###### Example: Vendor Data in a Profile
```
`#cloud-config
package_upgrade: true
packages:
- openssh-server
- curl
- htop
users:
- default
- name: admin
sudo: ALL=(ALL) NOPASSWD:ALL
ssh_authorized_keys:
- ssh-ed25519 AAAA... user@example.com
timezone: Europe/Amsterdam`
```
**Cloud-init runs only on first boot.** If you change cloud-init configuration after the instance has already started, the changes will not take effect. To apply new cloud-init config, you need to either recreate the instance or manually re-run cloud-init inside it.
**Merging limitation:** Cloud-init data is NOT merged across profiles. If two profiles both set cloud-init.vendor-data, the last profile wins entirely (the first profile's vendor-data is discarded, not merged).
##### 6.10 Advanced Config
The Advanced Config tab provides a **key-value pair editor** for any Incus configuration option not covered by the other tabs. This allows you to set custom or less common settings.
Each row has a **Key** field and a **Value** field. Click **Add Row** to add more entries.
Examples of keys you might set here:
- environment.HTTP_PROXY — Set proxy environment variables
- linux.kernel_modules — Kernel modules to load (containers)
- user.my-custom-key — Custom metadata (not interpreted by Incus)
- raw.lxc — Raw LXC configuration lines (containers, advanced use only)
- raw.qemu — Raw QEMU command-line arguments (VMs, advanced use only)
**Use with care:** The `raw.*` keys can break Incus in non-obvious ways. Only use them if you understand the underlying LXC or QEMU configuration and have a specific need not covered by the standard options.
### 7. Editing a Profile
Click the **Edit** button in the Actions column for any profile. This opens the same tabbed dialog as Create Profile, with all fields pre-populated with the current values.
Key differences from creation:
- **Profile Name** is read-only and cannot be changed.
- The **Advanced Config** tab shows config keys that are not mapped to form fields in the other tabs, plus any custom key-value pairs.
- Click **Save Changes** to apply your modifications.
**Changes propagate immediately!** When you save changes to a profile, they are applied to ALL instances using that profile. There is no confirmation step for this propagation. Be especially careful when editing profiles with high Used By counts.
### 8. Deleting a Profile
Click the **Delete** button in the Actions column. AETHER will ask for confirmation before deleting.
#### Deletion Rules
- **The `default` profile cannot be deleted.** It is a system profile required by Incus.
- **Profiles in use cannot be deleted.** If the Used By count is greater than 0, you must first remove the profile from all instances before deleting it.
- **Deletion is permanent.** The profile and all its configuration are removed. You would need to recreate it from scratch.
### 9. Common Profile Patterns
Profiles are most powerful when kept small and composable. Here are common patterns:
#### Environment-Based Profiles
Create separate profiles for different environments with appropriate resource levels:
|
Profile |
CPU |
Memory |
Root Disk |
Other Settings |
|
`res-dev` |
2 |
2 GiB |
20 GiB |
autostart: no |
|
`res-staging` |
4 |
8 GiB |
50 GiB |
autostart: yes |
|
`res-prod` |
8 |
16 GiB |
100 GiB |
autostart: yes, delete protection, daily snapshots |
#### Docker Host Profile
For containers that need to run Docker inside them:
- security.nesting = `true`
- security.syscalls.intercept.mknod = `true` (via Advanced Config)
- security.syscalls.intercept.setxattr = `true` (via Advanced Config)
#### Automated Snapshot Profile
Apply to any instance that needs regular backups:
- snapshots.schedule = `@daily`
- snapshots.expiry = `7d`
- snapshots.pattern = `daily-{{ creation_date|date:'2006-01-02' }}`
#### Cloud-Init Base Profile
Standard initial setup applied to all instances via vendor-data:
- Install common packages (openssh-server, curl, etc.)
- Create admin user with SSH key
- Set timezone
- Enable automatic security updates
#### Composable Stacking Example
A production web server might use multiple profiles stacked together:
Instance: web-prod-01
Profiles: [default, res-prod, snap-daily, cloud-init-base]
default -> root disk (/), eth0 (network)
res-prod -> 8 CPUs, 16 GiB memory, delete protection
snap-daily -> daily snapshots, 7-day retention
cloud-init-base -> admin user, packages, timezone
### 10. Profiles vs Instance Config
|
Scenario |
Use Profile |
Use Instance Config |
|
Network setup shared by many instances |
Yes |
|
|
Default root disk and storage pool |
Yes |
|
|
Standard resource limits for a class of instances |
Yes |
|
|
Snapshot schedules for all production instances |
Yes |
|
|
Cloud-init vendor-data (organization defaults) |
Yes |
|
|
Configuration that should propagate to many instances at once |
Yes |
|
|
Unique resource limits for one workload |
|
Yes |
|
Cloud-init user-data (per-instance setup) |
|
Yes |
|
GPU/USB passthrough for a specific instance |
|
Yes |
|
One-off or experimental settings |
|
Yes |
**Key principle:** Use profiles for shared, reusable configuration that many instances need. Use instance-specific config for unique, per-instance settings. Remember: editing a profile updates ALL instances using it.
### 11. Cluster Behavior
Profiles are **cluster-wide**. They are stored in the distributed database that is replicated across all cluster members.
- When you create or modify a profile on any cluster member, it is **immediately available on all members**.
- You do NOT need to manually sync profiles across members.
- Instances on any member can use any profile.
- Profiles can be pre-defined during cluster bootstrap using preseed files for consistent initial setup.
### 12. Best Practices
|
Practice |
Recommendation |
|
**Keep profiles small and composable** |
Create single-purpose profiles (one for networking, one for resource limits, one for snapshots, one for cloud-init) and stack them. This is more flexible than large monolithic profiles. |
|
**Use descriptive names** |
Name profiles to indicate their purpose: `net-bridged`, `res-large`, `snap-hourly`, `sec-docker`, `cloud-init-base`. |
|
**Add descriptions** |
Always fill in the Description field. It shows in the profile list and helps team members understand each profile's purpose. |
|
**Test before production** |
Before editing a profile used by production instances, test the changes on a development profile or instance first. Profile changes propagate immediately. |
|
**Use vendor-data in profiles** |
Put general cloud-init defaults (packages, users, timezone) in profile vendor-data. Put instance-specific setup in instance user-data. |
|
**Enable delete protection** |
Set security.protection.delete to `yes` in production profiles. This prevents accidental deletion of important instances. |
|
**Set boot priorities** |
For dependent services, use boot.autostart.priority and boot.autostart.delay to ensure infrastructure services (databases, DNS) start before application instances. |
|
**Don't modify default carelessly** |
The default profile is applied to all instances that don't specify profiles. Changes to it affect many instances. Prefer creating new profiles rather than heavily modifying the default. |
|
**Use --expanded to verify** |
After stacking multiple profiles on an instance, use `incus config show <instance> --expanded` in the CLI to verify the final effective configuration. |
---
## Incus – Settings
*Source: `/static/help_incus_settings.html`*
AETHER INCUS Server Settings Guide
## AETHER INCUS Server Settings Guide
This guide provides a comprehensive reference for all Incus server configuration settings available in the Settings tab of AETHER Infrastructure Management. It is intended to help administrators understand what each setting controls, its valid values, and the impact of changing it.
**Table of Contents**
- Understanding the Settings Tab
- Server Information Cards
- Cluster Settings
- Core Settings
- Images Settings
- Network & OVN Settings
- Storage Settings
- ACME (Let's Encrypt) Settings
- OpenID Connect (OIDC) Settings
- OpenFGA Settings
- Logging Settings
- Loki Settings (Legacy)
- Miscellaneous Settings
- User Custom Keys
### Understanding the Settings Tab
#### Global vs Local Scope
Each setting has a **scope** that determines how it is applied in a clustered environment:
|
Scope |
Meaning |
|
global |
The setting applies **cluster-wide**. Changing it on any member changes it for all members. Global settings control behaviour that must be consistent across the entire cluster (e.g., image replication count, offline threshold). |
|
local |
The setting applies **only to the specific cluster member** you are connected to. Each member can have a different value. Local settings are used for things that are member-specific (e.g., listen addresses, storage volumes). |
#### Configuration Table Columns
|
Column |
Description |
|
**Key** |
The configuration key name (e.g., `cluster.offline_threshold`). |
|
**Value** |
The current value. Values shown in *lighter text* indicate the setting is using its default value and has not been explicitly configured. |
|
**Description** |
A brief description of what the setting controls. |
|
**Scope** |
Whether the setting is global (cluster-wide) or local (per-member). |
|
**Default** |
The default value used when the setting has not been explicitly set. |
|
**Actions** |
The **Edit** button opens a modal where you can change the setting's value. The modal shows the type, scope, default, and description to help you make informed changes. |
#### Editing a Setting
Click **Edit** on any configuration key to open the edit modal. The modal displays the key name, its type (string, integer, bool), scope, current default, and description. Enter the new value and click **Save**.
**Tip:** To reset a setting to its default value, clear the value field entirely and save. An empty value tells Incus to fall back to the default.
**Warning:** Some settings (particularly networking and clustering settings) can affect cluster connectivity. Changing them incorrectly may make the cluster unreachable. Always verify the impact of a change before applying it.
### Server Information Cards
The top section of the Settings tab displays information cards showing key facts about the cluster member you are viewing:
|
Card |
Description |
|
**Server Name** |
The hostname of the cluster member. |
|
**Incus Version** |
The version of the Incus daemon running on this member. |
|
**OS** |
The operating system name and version (e.g., Ubuntu 24.04). |
|
**Kernel** |
The Linux kernel version running on this member. |
|
**Architecture** |
The CPU architecture (e.g., x86_64, aarch64). |
|
**Storage Driver** |
The storage backend and its version (e.g., zfs 2.2.x, lvm, btrfs). |
|
**Instance Driver** |
The virtualisation technology used to run instances (e.g., QEMU for VMs, LXC for containers). |
|
**Firewall** |
The firewall backend in use (e.g., nftables, xtables). |
|
**Clustered** |
Whether this server is part of a cluster (Yes/No). |
|
**CPU Threads** |
Total number of CPU threads available on this member. |
|
**Memory** |
Used and total RAM on this member, displayed with a usage bar. The bar colour changes from blue to yellow (>70%) to red (>85%) as usage increases. |
### Cluster Settings
These settings control how the Incus cluster operates, including failure detection, database quorum, image replication, and automatic workload rebalancing.
#### cluster.offline_threshold
| Property | Value |
| Type | integer |
| Default | `20` |
| Scope | global |
| Unit | seconds |
The number of seconds after which a cluster member that has not sent a heartbeat is considered **offline**. Once a member is marked offline, no operations can be performed on it, and its instances become unmanageable until it comes back online or is evacuated.
The minimum allowed value is **10 seconds**. Setting it too low may cause false positives during brief network hiccups. Setting it too high delays detection of genuinely failed members.
**Recommendation:** The default of 20 seconds works well for most environments. Only increase it if you experience transient network issues causing false offline detections. For high-availability environments, keep it low to enable fast failover.
#### cluster.healing_threshold
| Property | Value |
| Type | integer |
| Default | `0` (disabled) |
| Scope | global |
| Unit | seconds |
When set to a value greater than zero, Incus will **automatically evacuate** instances from a cluster member that has been offline for this many seconds. The instances are migrated to other healthy members using the configured evacuation mode.
Setting this to `0` (the default) disables automatic healing entirely — offline members must be manually evacuated or restored.
**Warning:** Enable this feature with care. If a member goes offline temporarily (e.g., during a reboot), automatic evacuation will move all its instances away. When the member comes back online, the instances will not automatically return. Set this value high enough (e.g., 300+ seconds) to avoid triggering during planned maintenance windows.
#### cluster.max_voters
| Property | Value |
| Type | integer |
| Default | `3` |
| Scope | global |
| Valid range | Odd number ≥ 3 |
The maximum number of cluster members assigned the **database voter** role. Voter members participate in the Raft distributed database consensus. The cluster requires a **majority of voters** (quorum) to be online for database operations to succeed.
- With `3` voters: cluster tolerates **1** voter failure
- With `5` voters: cluster tolerates **2** voter failures
- With `7` voters: cluster tolerates **3** voter failures
The value **must** be an odd number to ensure a clear majority can always be determined. Increasing voters improves fault tolerance but adds overhead to every database write since more members must acknowledge each operation.
**Recommendation:** 3 voters is ideal for small to medium clusters (3-10 members). Only increase to 5 for large clusters (10+ members) where the risk of losing 2 members simultaneously is realistic. Going beyond 5 is rarely needed.
#### cluster.max_standby
| Property | Value |
| Type | integer |
| Default | `2` |
| Scope | global |
| Valid range | 0 to 5 |
The maximum number of cluster members assigned the **database stand-by** role. Stand-by members maintain a copy of the database but do not vote. If a voter goes offline, a stand-by member is automatically **promoted to voter** to maintain quorum.
Members that are neither voters nor stand-by are called **spare** members — they run instances but do not participate in the database at all.
**Recommendation:** The default of 2 stand-by members provides good resilience. If a voter fails, one stand-by is promoted immediately, and another stand-by remains ready if a second failure occurs.
#### cluster.images_minimal_replica
| Property | Value |
| Type | integer |
| Default | `3` |
| Scope | global |
| Special values | `1` = no replication, `-1` = replicate to all members |
The minimum number of cluster members that keep a local copy of each image. When you add an image, Incus automatically replicates it to this many members, so instances can be launched locally without needing to transfer the image across the network first.
Setting this to `-1` replicates every image to every member, which uses more storage but ensures the fastest possible instance creation on any member.
#### cluster.join_token_expiry
| Property | Value |
| Type | string |
| Default | `3H` |
| Scope | global |
| Format | Duration string (e.g., `1H`, `30M`, `24H`) |
How long a cluster join token remains valid after it is generated. After this period, the token expires and a new one must be issued. This limits the window during which a stolen or leaked token can be used to join an unauthorised member to the cluster.
#### cluster.https_address
| Property | Value |
| Type | string |
| Default | - |
| Scope | local |
| Format | `IP:PORT` or `:PORT` |
The address this cluster member uses for internal cluster-to-cluster communication. This is typically the member's IP address with port 8443 (e.g., `192.168.1.10:8443`). Other cluster members use this address to reach this member for heartbeats, database replication, and image synchronisation.
**Critical:** Changing this incorrectly can make a cluster member unreachable by other members, effectively isolating it from the cluster. Only modify this if you are changing the member's network configuration.
#### Cluster Rebalancing Settings
These settings control the automatic workload rebalancing feature, which redistributes instances across cluster members when the load becomes uneven.
#### cluster.rebalance.interval
| Property | Value |
| Type | integer |
| Default | `0` (disabled) |
| Scope | global |
| Unit | minutes |
How often (in minutes) Incus evaluates whether instances should be moved between members to balance load. Setting this to `0` disables automatic rebalancing entirely.
#### cluster.rebalance.threshold
| Property | Value |
| Type | integer |
| Default | `20` |
| Scope | global |
| Unit | percentage |
The minimum **load difference** (as a percentage) between the most and least busy members required to trigger a migration. With the default of 20%, rebalancing only occurs when there is a significant load imbalance.
#### cluster.rebalance.batch
| Property | Value |
| Type | integer |
| Default | `1` |
| Scope | global |
The maximum number of instances that can be migrated in a single rebalancing run. A low value (like the default of 1) ensures rebalancing is gradual and does not cause excessive migration storms.
#### cluster.rebalance.cooldown
| Property | Value |
| Type | string |
| Default | `6H` |
| Scope | global |
| Format | Duration string (e.g., `1H`, `30M`) |
After an instance has been moved by the rebalancer, it will not be moved again for this duration. This prevents instances from being shuffled back and forth between members ("ping-pong" migrations).
### Core Settings
Core settings control fundamental server behaviour including API access, proxies, security, metrics, and daemon operation.
#### core.https_address
| Property | Value |
| Type | string |
| Default | - (not set, API not exposed) |
| Scope | local |
| Format | `IP:PORT` or `:PORT` |
The address and port where the Incus API listens for remote HTTPS connections. For example, `:8443` listens on all interfaces at port 8443, while `192.168.1.10:8443` listens only on that specific IP. If not set, the API is only accessible via the local Unix socket.
**Critical:** This is a local setting. In a cluster, each member must set its own `core.https_address` to be remotely accessible. Changing this to an incorrect value will make the API unreachable on that member.
#### core.metrics_address
| Property | Value |
| Type | string |
| Default | - (not set, metrics not exposed) |
| Scope | local |
| Format | `IP:PORT` or `:PORT` |
A separate HTTPS listen address for the Prometheus metrics endpoint (`/1.0/metrics`). Setting this allows you to expose metrics on a different port or interface than the main API, which can be useful for monitoring tools that should not have full API access.
#### core.metrics_authentication
| Property | Value |
| Type | bool |
| Default | `true` |
| Scope | global |
| Valid values | `true`, `false` |
Whether the metrics endpoint requires TLS client authentication. When `true`, only clients with a trusted certificate can scrape metrics. Set to `false` if your monitoring tool (e.g., Prometheus) cannot present a client certificate, but be aware this exposes metrics to anyone who can reach the metrics address.
#### core.proxy_http / core.proxy_https
| Property | Value |
| Type | string |
| Default | - (falls back to `HTTP_PROXY` / `HTTPS_PROXY` env vars) |
| Scope | global |
| Format | `http://proxy:port` or `http://user:pass@proxy:port` |
HTTP/HTTPS proxy servers for outbound connections made by the Incus daemon. This is used when Incus needs to reach external resources, such as remote image servers. If not set, Incus falls back to the standard `HTTP_PROXY` / `HTTPS_PROXY` environment variables.
#### core.proxy_ignore_hosts
| Property | Value |
| Type | string |
| Default | - (falls back to `NO_PROXY` env var) |
| Scope | global |
| Format | Comma-separated list (e.g., `1.2.3.4,internal.local`) |
Hosts that should bypass the proxy. Uses the same format as the `NO_PROXY` environment variable.
#### core.shutdown_timeout
| Property | Value |
| Type | integer |
| Default | `5` |
| Scope | global |
| Unit | minutes |
How many minutes the Incus daemon waits for running operations (e.g., live migrations, backups) to complete before forcing a shutdown. Increase this if you regularly perform long-running operations.
#### core.remote_token_expiry
| Property | Value |
| Type | string |
| Default | no expiry |
| Scope | global |
| Format | Duration string (e.g., `24H`, `7D`) |
How long a remote add token (used to add a new remote client) remains valid. By default, tokens do not expire. Setting an expiry is recommended for security.
#### core.trust_ca_certificates
| Property | Value |
| Type | bool |
| Default | `false` |
| Scope | global |
When `true`, any client presenting a certificate signed by the server's CA is automatically trusted. This enables certificate-based authentication without having to manually add each client certificate to the trust store.
#### core.https_trusted_proxy
| Property | Value |
| Type | string |
| Default | - |
| Scope | global |
| Format | Comma-separated IP addresses |
A list of trusted reverse proxy server IPs. When set, Incus uses the `X-Forwarded-For` header from these proxies to determine the real client IP address.
#### CORS Settings
The following settings control Cross-Origin Resource Sharing (CORS) headers. These are only needed if you access the Incus API directly from a web browser on a different domain.
|
Key |
Description |
|
`core.https_allowed_origin` |
Sets the `Access-Control-Allow-Origin` header. Use `*` for any origin, or a specific domain. |
|
`core.https_allowed_methods` |
Sets the `Access-Control-Allow-Methods` header (e.g., `GET, POST, PUT, DELETE`). |
|
`core.https_allowed_headers` |
Sets the `Access-Control-Allow-Headers` header. |
|
`core.https_allowed_credentials` |
Boolean. Sets `Access-Control-Allow-Credentials` to `true` when enabled. Default: `false`. |
#### BGP Settings
These settings enable the built-in BGP server for advertising network routes to external routers. BGP is used in advanced network setups where Incus networks need to be reachable from outside the cluster.
|
Key |
Scope |
Description |
|
`core.bgp_address` |
local |
Address to bind the BGP server to (e.g., `192.168.1.10:179`). |
|
`core.bgp_asn` |
global |
The BGP Autonomous System Number (ASN) for the Incus server. This identifies your network to BGP peers. |
|
`core.bgp_routerid` |
local |
A unique identifier for the BGP server on this member, formatted as an IPv4 address (e.g., `192.168.1.10`). Must be unique across all BGP-speaking members. |
#### Other Core Settings
|
Key |
Scope |
Default |
Description |
|
`core.dns_address` |
local |
- |
Address to bind the built-in authoritative DNS server for instance DNS records. |
|
`core.debug_address` |
local |
- |
Address to bind the Go `pprof` debug server (HTTP). Used for performance profiling. Do not expose in production. |
|
`core.storage_buckets_address` |
local |
- |
Address to bind the S3-compatible storage object server (HTTPS). Required for accessing storage buckets via the S3 protocol. |
|
`core.syslog_socket` |
local |
`false` |
Enable a Unix syslog socket that external processes can use to send log messages to the Incus log. |
### Images Settings
These settings control how Incus manages images — particularly cached images that are fetched from remote servers.
#### images.auto_update_cached
| Property | Value |
| Type | bool |
| Default | `true` |
| Scope | global |
Whether Incus should automatically check for and download newer versions of cached images. When `true`, Incus periodically contacts the remote server each image was fetched from and downloads updates if available. Existing running instances are not affected — only new instances launched from the image will use the updated version.
#### images.auto_update_interval
| Property | Value |
| Type | integer |
| Default | `6` |
| Scope | global |
| Unit | hours |
How often (in hours) Incus checks for updated cached images. Set to `0` to disable automatic update checks entirely. Lower values increase network traffic to remote image servers but ensure you have the latest security patches sooner.
#### images.remote_cache_expiry
| Property | Value |
| Type | integer |
| Default | `10` |
| Scope | global |
| Unit | days |
The number of days an unused cached image is kept before being automatically removed. Each time an instance is launched from a cached image, the "last used" timestamp resets. Images that remain unused for this many days are flushed to save storage.
**Tip:** If you frequently create short-lived instances from the same image with gaps longer than 10 days between uses, consider increasing this value to avoid repeatedly re-downloading the image.
#### images.compression_algorithm
| Property | Value |
| Type | string |
| Default | `gzip` |
| Scope | global |
| Valid values | `bzip2`, `gzip`, `lz4`, `lzma`, `xz`, `zstd`, `none` |
The compression algorithm used when creating new images (e.g., from publishing an instance). Affects the trade-off between image size and compression/decompression speed:
| Algorithm | Speed | Compression ratio | Notes |
| `lz4` | Fastest | Lower | Best for environments where speed matters more than size |
| `zstd` | Fast | Good | Good balance of speed and compression |
| `gzip` | Medium | Good | Widely compatible default |
| `xz` / `lzma` | Slow | Best | Smallest files, but slow to compress |
| `bzip2` | Slow | Good | Legacy option, generally outperformed by zstd |
| `none` | Instant | None | No compression — fastest but largest files |
#### images.default_architecture
| Property | Value |
| Type | string |
| Default | - (auto-detected) |
| Scope | global |
In mixed-architecture clusters (e.g., some members are x86_64 and others are aarch64), this specifies the default architecture to use when the architecture is not explicitly specified for an operation.
### Network & OVN Settings
These settings configure how Incus connects to OVN (Open Virtual Network) and OVS (Open vSwitch) for software-defined networking.
#### network.ovn.northbound_connection
| Property | Value |
| Type | string |
| Default | `unix:/run/ovn/ovnnb_db.sock` |
| Scope | global |
| Format | `unix:/path` or `ssl:IP:PORT` or `tcp:IP:PORT` |
The connection string to the OVN Northbound database. The Northbound database stores the logical network configuration (switches, routers, ports, ACLs). Incus sends all OVN network configuration changes to this database.
For a single-node setup, the default Unix socket works. For a multi-node OVN cluster, use SSL or TCP connections to the Northbound leader, e.g., `ssl:10.0.0.1:6641,ssl:10.0.0.2:6641,ssl:10.0.0.3:6641`.
#### network.ovn.integration_bridge
| Property | Value |
| Type | string |
| Default | `br-int` |
| Scope | global |
The name of the OVS integration bridge that OVN uses. This bridge is automatically managed by OVN and is where instance network ports are connected. You almost never need to change this from the default.
#### OVN SSL Certificates
When connecting to the OVN Northbound database over SSL, these settings provide the TLS certificates:
|
Key |
Default |
Description |
|
`network.ovn.ca_cert` |
Content of `/etc/ovn/ovn-central.crt` |
The CA certificate that signed the OVN server certificate. |
|
`network.ovn.client_cert` |
Content of `/etc/ovn/cert_host` |
The client certificate Incus presents when connecting to OVN. |
|
`network.ovn.client_key` |
Content of `/etc/ovn/key_host` |
The private key for the client certificate. |
#### network.ovs.connection
| Property | Value |
| Type | string |
| Default | `unix:/run/openvswitch/db.sock` |
| Scope | global |
The path to the local OVS database socket. OVS (Open vSwitch) is the underlying datapath engine that OVN uses. This setting should rarely need to be changed.
#### network.hwaddr_pattern
| Property | Value |
| Type | string |
| Default | `10:66:6a:xx:xx:xx` |
| Scope | global |
A template for generating MAC addresses for instance network interfaces. Every `x` in the pattern is replaced with a random hexadecimal character (`0`-`f`). The first three octets (e.g., `10:66:6a`) form the OUI (Organizationally Unique Identifier) and should be kept consistent within your cluster.
**Warning:** Be aware of MAC address collisions (the "birthday paradox"). With three random octets (the default `xx:xx:xx`), collisions become likely after around 4,000 instances. If you run very large deployments, consider using a pattern with more random characters.
### Storage Settings
#### storage.backups_volume
| Property | Value |
| Type | string |
| Default | - (stored in the daemon directory) |
| Scope | local |
| Format | `POOL/VOLUME` (e.g., `default/backups`) |
Specifies a storage volume to store instance backup tarballs. By default, backups are stored in the Incus daemon directory on the root filesystem. Setting this moves backup storage to a dedicated storage volume, which is recommended in production to avoid filling the root filesystem.
#### storage.images_volume
| Property | Value |
| Type | string |
| Default | - (stored in the daemon directory) |
| Scope | local |
| Format | `POOL/VOLUME` (e.g., `default/images`) |
Specifies a storage volume to store image tarballs. Similar to `storage.backups_volume`, this moves image storage off the root filesystem.
#### LINSTOR Storage Settings
These settings configure connectivity to a LINSTOR storage backend, which provides distributed block storage for Incus.
|
Key |
Description |
|
`storage.linstor.controller_connection` |
Connection string to the LINSTOR controller. |
|
`storage.linstor.ca_cert` |
CA certificate for SSL connections to the LINSTOR controller. |
|
`storage.linstor.client_cert` |
Client certificate for SSL authentication to the LINSTOR controller. |
|
`storage.linstor.client_key` |
Client private key for SSL authentication. |
|
`storage.linstor.satellite.name` |
Override for the local LINSTOR satellite node name, if it differs from the Incus server name. |
### ACME (Let's Encrypt) Settings
ACME (Automatic Certificate Management Environment) allows Incus to automatically obtain and renew TLS certificates from Let's Encrypt or another ACME-compatible CA. This replaces the self-signed certificate that Incus uses by default.
|
Key |
Type |
Default |
Description |
|
`acme.agree_tos` |
bool |
`false` |
You must set this to `true` to agree to the ACME provider's Terms of Service before certificates can be issued. |
|
`acme.domain` |
string |
- |
The domain name for which the certificate should be issued (e.g., `incus.example.com`). This domain must resolve to the Incus server. |
|
`acme.email` |
string |
- |
Email address for the ACME account. Used by the CA to send expiry notifications. |
|
`acme.ca_url` |
string |
`https://acme-v02.api.letsencrypt.org/directory` |
The ACME directory URL. Change this to use a different CA (e.g., a staging environment for testing: `https://acme-staging-v02.api.letsencrypt.org/directory`). |
|
`acme.challenge` |
string |
`HTTP-01` |
The challenge type used to prove domain ownership. `HTTP-01` requires port 80 to be reachable. `DNS-01` uses DNS records (requires a supported DNS provider). |
|
`acme.http.port` |
string |
`:80` |
The address and port to listen on for HTTP-01 challenges. Only used when `acme.challenge` is `HTTP-01`. |
|
`acme.provider` |
string |
- |
The DNS provider backend for DNS-01 challenges (e.g., `cloudflare`, `route53`). Only used when `acme.challenge` is `DNS-01`. |
|
`acme.provider.environment` |
string |
- |
Environment variables passed to the DNS provider (e.g., API keys). Format depends on the provider. |
|
`acme.provider.resolvers` |
string |
- |
Comma-separated list of DNS resolvers used during DNS-01 challenges for CNAME resolution and apex domain determination. |
### OpenID Connect (OIDC) Settings
These settings enable authentication via an external OpenID Connect identity provider (e.g., Keycloak, Auth0, Okta, Google). When configured, users can authenticate to the Incus API using their organisational identity.
|
Key |
Type |
Description |
|
`oidc.issuer` |
string |
The OpenID Connect Discovery URL for the identity provider (e.g., `https://auth.example.com/realms/incus`). Incus uses this to discover the provider's endpoints. |
|
`oidc.client.id` |
string |
The client ID registered with the identity provider for Incus. |
|
`oidc.audience` |
string |
The expected audience claim in the token. Required by some providers to validate that the token was intended for Incus. |
|
`oidc.claim` |
string |
Which claim in the OIDC token to use as the username in Incus (e.g., `email`, `preferred_username`). |
|
`oidc.scopes` |
string |
Comma-separated list of OAuth2 scopes to request (e.g., `openid,profile,email`). |
### OpenFGA Settings
OpenFGA is a fine-grained authorization system. When configured, Incus uses OpenFGA to control who can access which resources (projects, instances, etc.) with granular permissions.
|
Key |
Type |
Description |
|
`openfga.api.url` |
string |
The URL of the OpenFGA server (e.g., `http://openfga.internal:8080`). |
|
`openfga.api.token` |
string |
The API authentication token for the OpenFGA server. |
|
`openfga.store.id` |
string |
The ID of the OpenFGA permission store that contains the authorization model for Incus. |
### Logging Settings
The logging framework supports sending Incus events to external logging services. You can configure multiple named targets (replace `NAME` with your chosen identifier, e.g., `logging.mysyslog.target.type`).
#### Target Configuration
|
Key |
Type |
Default |
Description |
|
`logging.NAME.target.type` |
string |
- |
The type of logging target. Valid values: `loki` (Grafana Loki), `syslog` (RFC 5424 syslog), `webhook` (HTTP POST). |
|
`logging.NAME.target.address` |
string |
- |
The address of the logging server. Include protocol, host, and port (e.g., `tcp://syslog.example.com:514` or `https://loki.example.com:3100`). |
|
`logging.NAME.target.username` |
string |
- |
Username for authentication with the logging server. |
|
`logging.NAME.target.password` |
string |
- |
Password for authentication with the logging server. |
|
`logging.NAME.target.ca_cert` |
string |
- |
CA certificate for verifying the logging server's TLS certificate. |
|
`logging.NAME.target.facility` |
string |
- |
The syslog facility (only for syslog targets). Common values: `local0` through `local7`, `daemon`, `user`. |
|
`logging.NAME.target.instance` |
string |
hostname |
Custom name for the "instance" field in Loki events. Useful for identifying the cluster rather than individual members. |
|
`logging.NAME.target.labels` |
string |
- |
Comma-separated labels for Loki log entries (e.g., `environment,cluster`). |
|
`logging.NAME.target.retry` |
integer |
`3` |
Number of delivery retries if sending a log entry fails. |
#### Event Filtering
|
Key |
Type |
Default |
Description |
|
`logging.NAME.types` |
string |
`lifecycle,logging` |
Which event types to send. Valid values (comma-separated): `lifecycle` (instance/resource events), `logging` (daemon log entries), `network-acl` (network ACL traffic logs). |
|
`logging.NAME.logging.level` |
string |
`info` |
Minimum log level to forward. Valid values: `debug`, `info`, `warn`, `error`. |
|
`logging.NAME.lifecycle.types` |
string |
- (all types) |
Filter lifecycle events to specific resource types (e.g., `instance`, `image`, `network`). Empty means all types. |
|
`logging.NAME.lifecycle.projects` |
string |
- (all projects) |
Filter lifecycle events to specific projects. Empty means all projects. |
### Loki Settings (Legacy)
These settings are the **legacy** method for configuring Grafana Loki integration. For new deployments, use the `logging.NAME.*` settings above instead, which support multiple targets and additional target types.
|
Key |
Type |
Default |
Description |
|
`loki.api.url` |
string |
- |
URL to the Loki server (e.g., `https://loki.example.com:3100`). Incus appends `/loki/api/v1/push` automatically. |
|
`loki.api.ca_cert` |
string |
- |
CA certificate for verifying the Loki server's TLS certificate. |
|
`loki.auth.username` |
string |
- |
Username for Loki authentication. |
|
`loki.auth.password` |
string |
- |
Password for Loki authentication. |
|
`loki.loglevel` |
string |
`info` |
Minimum log level to send. Valid values: `debug`, `info`, `warn`, `error`. |
|
`loki.types` |
string |
`lifecycle,logging` |
Event types to send. Valid values: `lifecycle`, `logging`, `network-acl`. |
|
`loki.labels` |
string |
- |
Comma-separated labels for Loki log entries. |
|
`loki.instance` |
string |
hostname |
Custom name for the "instance" field in Loki events. |
### Miscellaneous Settings
#### backups.compression_algorithm
| Property | Value |
| Type | string |
| Default | `gzip` |
| Scope | global |
| Valid values | `bzip2`, `gzip`, `lz4`, `lzma`, `xz`, `zstd`, `none` |
The compression algorithm used for instance backups (created via `incus export` or the API). Same trade-offs as `images.compression_algorithm` — see the Images Settings section for a comparison table.
#### instances.nic.host_name
| Property | Value |
| Type | string |
| Default | `random` |
| Scope | global |
| Valid values | `random`, `mac` |
How the host-side network interface name is generated for instance NICs:
- `random` — Uses a random interface name (e.g., `vethXXXXXX`). This is the default and works well in most cases.
- `mac` — Generates the name from the MAC address (e.g., `inc1066XXXXXX`). This makes it easier to correlate host interfaces with specific instances.
#### instances.lxcfs.per_instance
| Property | Value |
| Type | bool |
| Default | `false` |
| Scope | global |
LXCFS provides overlays for `/proc` and `/sys` inside containers so that tools like `free`, `top`, and `nproc` show the container's resource limits rather than the host's total resources. By default, a single LXCFS instance serves all containers. Setting this to `true` runs a separate LXCFS per container, which uses more resources but isolates LXCFS crashes to individual containers.
#### instances.placement.scriptlet
| Property | Value |
| Type | string |
| Default | - |
| Scope | global |
A Starlark (Python-like) script that controls where new instances are placed in the cluster. This allows custom placement logic beyond the built-in scheduler. The scriptlet receives information about the requested instance and available cluster members, and returns the target member.
#### authorization.scriptlet
| Property | Value |
| Type | string |
| Default | - |
| Scope | global |
A Starlark script for custom authorization logic. When set, Incus calls this script for every API request to determine whether the request should be allowed. This provides maximum flexibility for complex authorization requirements.
### User Custom Keys
#### user.*
| Property | Value |
| Type | string |
| Scope | global |
Free-form key/value pairs for your own use. Incus does not interpret these settings — they are purely for storing custom metadata on the server. For example, you could use `user.environment=production` or `user.contact=admin@example.com`.
Keys under `user.ui.*` are special: they are visible to **unauthenticated** users. This can be used to provide information to UI clients before authentication.
---
## Incus – Images
*Source: `/static/help_incus_images.html`*
AETHER INCUS Images Guide
## AETHER INCUS Images Guide
This guide explains how Incus images work and how to use the Images tab in AETHER Infrastructure Management.
### What Are Incus Images?
An Incus image is a packaged template used to create new instances (containers or virtual machines). Images contain the root filesystem and metadata needed to launch an instance. Think of an image as a blueprint: every time you create a new instance from an image, you get a fresh copy of that filesystem.
Images can come from several sources:
- **Remote image servers** — Public repositories such as the official Incus image server provide pre-built images for common Linux distributions (Ubuntu, Debian, Alpine, etc.).
- **Manual upload** — You can upload your own custom images (e.g., a golden image with your applications pre-installed).
- **Publishing an instance** — A running instance can be published as an image, allowing you to clone or distribute it.
### Image Types
Each image is one of two types:
|
Type |
Description |
|
**Container** |
A lightweight OS-level image. Container instances share the host kernel and start in seconds. Best for workloads that do not require a dedicated kernel. |
|
**Virtual Machine** |
A full machine image including its own kernel. VM instances run in complete isolation with their own kernel, similar to traditional virtualisation. Use this when you need kernel-level control or must run a different OS (e.g., Windows). |
### Fingerprints
Every image is uniquely identified by its **fingerprint** — a SHA256 hash of the image contents. The fingerprint is a 64-character hexadecimal string, for example:
```
`a3f5b8c9d1e2f4a6b8c0d2e4f6a8b0c2d4e6f8a0b2c4d6e8f0a2b4c6d8e0f2`
```
In the images table, fingerprints are shortened to the first 12 characters for readability. The full fingerprint is visible in the image detail view.
**Note:** Because the fingerprint is derived from the image contents, two identical images will always have the same fingerprint regardless of where they are stored.
### Aliases
While fingerprints uniquely identify images, they are not human-friendly. **Aliases** let you assign memorable names to images. An alias is simply a label that points to a specific image fingerprint.
For example, instead of referencing an image by `a3f5b8c9d1e2...`, you can create an alias like `ubuntu-24.04` or `my-webapp-base`.
#### Key Points About Aliases
- An image can have **multiple aliases** (e.g., `ubuntu-24.04` and `ubuntu-noble` pointing to the same image).
- An alias can only point to **one image** at a time.
- Aliases are used when launching instances — you can specify an alias name instead of a fingerprint.
- Each alias has an optional **description** field for documentation.
- You can add or remove aliases from an image at any time via the **View/Edit** modal.
**Tip:** Use descriptive aliases that include version information (e.g., `ubuntu-24.04-v2`) so your team knows exactly which image they are launching.
### The Public Setting
The **Public** boolean controls whether the image is accessible to untrusted clients:
|
Value |
Behaviour |
|
**No** (default) |
The image is *private*. Only authenticated and trusted clients connected to this Incus server can see and use it. |
|
**Yes** |
The image is *public*. Any client that can reach this Incus server — including untrusted clients — can list and download the image. This is how public image servers make their images available. |
**Warning:** Only set an image to Public if you intentionally want it available to anyone who can connect to your Incus cluster. For most internal use cases, images should remain private.
### The Auto-Update Setting
The **Auto-Update** boolean determines whether Incus should automatically check for and download newer versions of this image from its original source:
|
Value |
Behaviour |
|
**No** (default) |
The image stays as-is. It will not be updated automatically, even if a newer version becomes available at the source. |
|
**Yes** |
Incus periodically checks the original remote server for an updated version of this image. If a newer version is found, Incus downloads it and replaces the local copy. Existing running instances are *not* affected — only new instances launched from the image will use the updated version. |
**Note:** Auto-Update only works for images that were originally fetched from a remote image server. Manually uploaded images have no remote source to check, so enabling Auto-Update on them has no effect. When an image has a known update source, it is displayed in the View/Edit detail modal.
### Cached vs Non-Cached Images
The **Cached** column indicates how the image arrived on the cluster:
|
Value |
Meaning |
|
**No** |
The image was explicitly added — either manually uploaded, copied from another server, or intentionally downloaded. These images persist until you delete them. |
|
**Yes** |
The image was automatically cached by Incus when an instance was launched from a remote image. Cached images may be automatically cleaned up by Incus if they are not used within the configured expiry period (default: 10 days). |
#### Download Button Behaviour
|
Button |
Condition |
Reason |
|
Download |
Non-cached image |
The image is explicitly stored on the cluster. You can download it as a backup or to transfer it elsewhere. |
|
Download |
Cached image |
The image is a temporary cache entry that can be re-fetched from its remote source. Downloading it is disabled. |
### Uploading Images
Use the **Upload Image** button to add a custom image to the cluster. The upload form has the following fields:
|
Field |
Required |
Description |
|
**Image File** |
Yes |
The image archive file. Supported formats are `.tar.gz`, `.tar.xz`, `.tgz`, and `.txz`. This file should contain the root filesystem and image metadata as produced by Incus image export or compatible image build tools (e.g., distrobuilder). |
|
**Alias** |
No |
An optional alias name to assign to the uploaded image. If omitted, the image will only be identifiable by its fingerprint. |
|
**Public** |
No |
Whether to make the uploaded image public (see the Public setting section above). Defaults to unchecked (private). |
**Tip:** A progress bar shows the upload progress. Large images may take some time to upload depending on your connection speed.
### Images Table Columns
The images table displays the following information for each image:
|
Column |
Description |
|
**Fingerprint** |
The first 12 characters of the image's SHA256 fingerprint. Click View/Edit to see the full fingerprint. |
|
**Aliases** |
Human-readable names assigned to this image. An image may have zero, one, or multiple aliases. |
|
**Description** |
A text description of the image, typically set by the image creator (e.g., "Ubuntu 24.04 LTS"). |
|
**Type** |
Whether the image is a Container or Virtual Machine image. |
|
**Architecture** |
The CPU architecture the image is built for (e.g., x86_64, aarch64). |
|
**Size** |
The compressed size of the image on disk. |
|
**Auto-Update** |
Whether Incus will automatically fetch newer versions from the remote source. |
|
**Cached** |
Whether the image was auto-cached from a remote server or explicitly added. |
|
**Upload Date** |
The date the image was added to this cluster. |
|
**Actions** |
The **View/Edit** button opens a detail modal where you can view full image properties, modify settings (Public, Auto-Update), manage aliases, refresh the image from source, or delete the image. |
|
**Download** |
Download the image file. Only available for non-cached images (see Cached vs Non-Cached above). |
### Managing Images (View/Edit)
Clicking **View/Edit** on any image opens a detail modal with the following capabilities:
- **View details** — Full fingerprint, type, architecture, size, creation date, upload date, and last used date.
- **Change Public setting** — Toggle whether the image is publicly accessible.
- **Change Auto-Update setting** — Enable or disable automatic updates from the remote source.
- **Manage aliases** — Add new aliases or remove existing ones.
- **View properties** — See all key-value metadata properties set on the image.
- **Refresh Image** — For images with a known remote source, manually trigger an update check. This button only appears when an update source is available.
- **Delete Image** — Permanently remove the image from the cluster. A confirmation prompt is shown before deletion.
**Warning:** Deleting an image does not affect instances that were already created from it. However, you will no longer be able to create new instances from the deleted image.
---
## Incus – Configuration
*Source: `/static/help_incus_configuration.html`*
AETHER INCUS Project Configuration Guide
## AETHER INCUS Project Configuration Guide
This guide explains Incus project configuration — how projects provide isolation and resource limits, what each setting controls, and how to configure them in AETHER Infrastructure Management.
#### Table of Contents
- 1. What Are Incus Projects?
- 2. The Configuration Tab Overview
- 3. Project Information
- 4. Feature Flags
- 5. Resource Limits & Usage
- 6. The "Restricted" Toggle
- 7. Restriction Categories (CLI)
- 7.1 Container Restrictions
- 7.2 Virtual Machine Restrictions
- 7.3 Device Restrictions
- 7.4 Network Restrictions
- 7.5 Cluster Restrictions
- 7.6 Backup & Snapshot Restrictions
- 7.7 ID Mapping Restrictions
- 8. Saving Configuration
- 9. Common Use Cases
- 10. Best Practices
### 1. What Are Incus Projects?
Projects are **organizational containers** in Incus that group related instances (containers and VMs) together. They serve two primary purposes:
- **Organization** — Group instances by purpose, customer, environment, or team. Instance names can be reused across different projects.
- **Isolation & resource control** — In multi-user environments, confine each user to their own project with resource limits and restrictions, preventing access to other users' resources.
#### The Default Project
Every Incus installation has a `default` project. All instances belong to the default project unless you create and switch to another project. The default project always exists and cannot be deleted.
#### How Isolation Works
Instances are **always isolated** between projects — they cannot be shared, and instance names can be duplicated across projects. Other entities (images, profiles, networks, storage volumes, storage buckets) can either be **isolated within the project** or **inherited from the default project**, controlled by Feature Flags.
Project Isolation Model:
┌─ Default Project ────────────────────┐
│ Instances: web-01, db-01 │
│ Images: ubuntu-24.04, debian-12 │ <── Shared if features.images=false
│ Profiles: default, web-server │ <── Shared if features.profiles=false
│ Networks: incusbr0, ovn-prod │ <── Shared if features.networks=false
│ Storage Volumes: data-vol │ <── Shared if features.storage.volumes=false
└──────────────────────────────────────┘
┌─ Project "dev-team" ─────────────────┐
│ Instances: web-01, db-01 │ <── Always isolated (same names OK)
│ Feature flags control whether │
│ other entities are isolated or │
│ inherited from default project │
│ Resource limits control capacity │
└──────────────────────────────────────┘
### 2. The Configuration Tab Overview
The Configuration tab in AETHER displays and manages the project configuration for the currently connected cluster. It is organized into three sections:
|
Section |
Purpose |
|
**Project Information** |
View the project name and edit its description. |
|
**Feature Flags** |
Toggle switches that control which entities (images, profiles, networks, storage) are isolated within this project or inherited from the default project. |
|
**Resource Limits & Usage** |
Set maximum resource quotas and view current usage for instances, CPU, memory, disk, networks, and processes. |
After making changes, click **Save Configuration** at the bottom to apply them.
### 3. Project Information
|
Field |
Editable |
Description |
|
**Project Name** |
No (read-only) |
The name of the current project. This is set when the project is created and cannot be changed here. |
|
**Description** |
Yes |
A human-readable description of the project's purpose. Use this to document what the project is for, who uses it, or what environment it represents. |
### 4. Feature Flags
Feature flags control **which entities are isolated** within this project versus inherited from the default project. Each flag is a toggle switch:
- **On (green)** — The entity is **isolated**. This project has its own separate set, independent from the default project.
- **Off (gray)** — The entity is **inherited**. This project shares the entity from the default project.
**Feature flags lock after instances are added!** Once a project contains instances, feature flags cannot be changed. You must remove all instances from the project first before modifying feature flags. Set your desired features **before** creating any instances in the project.
|
Toggle |
Config Key |
Description |
|
**Images** |
features.images |
When **on**, this project has its own set of images and image aliases. Images imported or cached in this project are not visible in other projects, and vice versa. When **off**, the project shares images from the default project. |
|
**Networks** |
features.networks |
When **on**, this project has its own set of networks (bridges, OVN networks, etc.). When **off**, the project uses networks from the default project. Not enabled by default — only turn on if you need network isolation between projects. |
|
**Network Zones** |
features.networks.zones |
When **on**, this project has its own DNS network zones. Zones provide a project-oriented "view" of addresses — only instances in this project are included. Zone names must be globally unique even across projects. |
|
**Profiles** |
features.profiles |
When **on**, this project has its own set of profiles, separate from the default project's profiles. **Important:** When isolated, the project's own `default` profile starts empty — you must configure it with a root disk and network device, or instance creation will fail. |
|
**Storage Volumes** |
features.storage.volumes |
When **on**, this project has its own set of custom storage volumes. When **off**, custom volumes are shared from the default project. |
|
**Storage Buckets** |
features.storage.buckets |
When **on**, this project has its own set of S3-compatible storage buckets. When **off**, buckets are shared from the default project. |
|
**Restricted** |
restricted |
Master toggle for security restrictions. When **on**, security-sensitive features are blocked by default (privileged containers, raw config, device passthrough, etc.). Individual restrictions can be relaxed via CLI. See Section 6 for details. |
**Profiles isolation caveat:** When you enable features.profiles, the project gets its own empty `default` profile. Unlike the system default profile, this one has no root disk or network device. You must add them manually (e.g., via `incus profile edit default` within the project), or instances created in this project will fail to start.
### 5. Resource Limits & Usage
Resource limits set **maximum quotas** for the project. The table shows four columns for each resource:
|
Column |
Description |
|
**Resource** |
The type of resource being limited. |
|
**Limit** |
The maximum allowed value. Leave empty for **unlimited** (no quota). Enter a number or size value to set a cap. |
|
**Current Usage** |
How much of this resource is currently allocated by instances in the project. |
|
**Usage %** |
Current usage as a percentage of the limit. Shows "UNLIMITED" when no limit is set. |
#### Available Resource Limits
|
Resource |
Config Key |
Type |
Description |
|
**instances** |
limits.instances |
Integer |
Maximum total number of instances (containers + VMs combined) allowed in the project. |
|
**containers** |
limits.containers |
Integer |
Maximum number of containers allowed. Does not count VMs. |
|
**virtual-machines** |
limits.virtual-machines |
Integer |
Maximum number of virtual machines allowed. Does not count containers. |
|
**cpu** |
limits.cpu |
Integer |
Maximum total CPUs that can be allocated across all instances. This is the **sum of all instance `limits.cpu` values**, not actual CPU usage. Every instance must have `limits.cpu` set when this limit is active. |
|
**memory** |
limits.memory |
Size |
Maximum total memory allocatable across all instances (e.g., `50GiB`). This is the **sum of all instance `limits.memory` values**, not actual memory usage. Must be an absolute value, not a percentage. |
|
**disk** |
limits.disk |
Size |
Maximum total disk space across all instance volumes, custom volumes, and images in the project (e.g., `500GiB`). |
|
**networks** |
limits.networks |
Integer |
Maximum number of networks that can be created in the project. |
|
**processes** |
limits.processes |
Integer |
Maximum total processes across all instances (sum of instance `limits.processes` values). |
**How limits work:** Resource limits apply to **configured allocations** (instance config values), not actual runtime usage. For example, if limits.memory is set to `50GiB`, the sum of all instance `limits.memory` settings cannot exceed 50 GiB, regardless of how much memory is actually being used at any moment. This prevents overcommitment at the configuration level.
**CPU and memory requirements:** When limits.cpu is set, every instance in the project must have `limits.cpu` configured, and CPU pinning (assigning specific core numbers) is not allowed. When limits.memory is set, every instance must have `limits.memory` set to an absolute value (not a percentage).
#### Valid Size Suffixes
For memory and disk limits, use these suffixes:
|
Binary (recommended) |
Decimal |
| KiB (1024 bytes) | kB (1000 bytes) |
| MiB | MB |
| GiB | GB |
| TiB | TB |
### 6. The "Restricted" Toggle
The **Restricted** toggle (restricted) is a master switch for security restrictions. When enabled, it **blocks access to security-sensitive features by default**, such as:
- Running privileged containers
- Using raw LXC or QEMU configuration
- Passing through GPUs, USB devices, PCI devices
- Mounting arbitrary host paths
- Creating backups and snapshots
- Targeting specific cluster members
- Enabling container nesting (Docker-in-Incus)
This is essential for **multi-user environments** where you want to prevent project users from escalating privileges or accessing host resources they shouldn't have.
**Granular control via CLI:** While AETHER provides the master restricted toggle, individual restriction categories can be fine-tuned via the Incus CLI. For example, you can enable restrictions overall but allow nesting for Docker use cases:
```
`incus project set my-project restricted=true
incus project set my-project restricted.containers.nesting=allow`
```
### 7. Restriction Categories (CLI)
When restricted=true, the following restriction categories take effect. Each defaults to the most secure setting (`block`) unless explicitly relaxed. These are configured via the Incus CLI using `incus project set`.
##### 7.1 Container Restrictions
|
Key |
Default |
Values |
Description |
|
restricted.containers.nesting |
block |
allow, block |
Controls whether `security.nesting=true` can be set. Required for running Docker or Incus inside containers. |
|
restricted.containers.lowlevel |
block |
allow, block |
Controls access to low-level config: `raw.lxc`, `raw.idmap`, and `volatile.*` keys. These can bypass security boundaries. |
|
restricted.containers.privilege |
unprivileged |
unprivileged, isolated, allow |
**unprivileged:** Containers must be unprivileged (no `security.privileged=true`). **isolated:** Additionally prevents `security.idmap.isolated=true`. **allow:** No restriction on privilege settings. |
|
restricted.containers.interception |
block |
allow, block, full |
**block:** All syscall interception forbidden. **allow:** Safe interception allowed (mknod, bpf, setxattr), mount stays blocked. **full:** All interception including mount (use with caution). |
##### 7.2 Virtual Machine Restrictions
|
Key |
Default |
Values |
Description |
|
restricted.virtual-machines.lowlevel |
block |
allow, block |
Controls access to raw QEMU configuration: `raw.qemu`, `raw.qemu.conf`, and QMP command keys. These can be used to bypass VM isolation. |
##### 7.3 Device Restrictions
|
Key |
Default |
Values |
Description |
|
restricted.devices.disk |
managed |
allow, block, managed |
**block:** All disks forbidden except root. **managed:** Only Incus pool-backed disks allowed (`pool=` required). **allow:** Any disk source, but paths checked against `restricted.devices.disk.paths`. |
|
restricted.devices.disk.paths |
(empty) |
Comma-separated paths |
When disk=allow, restricts which host source paths are permitted. Example: `/mnt/shared,/srv/data`. If empty, all paths allowed. |
|
restricted.devices.nic |
managed |
allow, block, managed |
**block:** All NICs forbidden. **managed:** Only managed network NICs allowed (`network=` required). **allow:** Any NIC type including direct host interface access. |
|
restricted.devices.gpu |
block |
allow, block |
Controls GPU passthrough. When blocked, no GPU devices can be added to instances. |
|
restricted.devices.usb |
block |
allow, block |
Controls USB device passthrough. |
|
restricted.devices.pci |
block |
allow, block |
Controls direct PCI device passthrough. |
|
restricted.devices.proxy |
block |
allow, block |
Controls proxy devices (port forwarding between host and instance). |
|
restricted.devices.infiniband |
block |
allow, block |
Controls InfiniBand device passthrough. |
|
restricted.devices.unix-char |
block |
allow, block |
Controls Unix character device passthrough (e.g., `/dev/ttyUSB0`). |
|
restricted.devices.unix-block |
block |
allow, block |
Controls Unix block device passthrough. |
|
restricted.devices.unix-hotplug |
block |
allow, block |
Controls dynamic Unix device attachment. |
##### 7.4 Network Restrictions
|
Key |
Default |
Values |
Description |
|
restricted.networks.access |
(empty) |
Comma-separated names |
Specifies which managed networks are available for use in this project. If empty, all networks are accessible. |
|
restricted.networks.uplinks |
(empty) |
Comma-separated names |
Specifies which networks can be used as uplinks for OVN networks in this project. |
|
restricted.networks.subnets |
block |
Comma-separated uplink:subnet pairs |
Specifies which subnets from uplink networks are allocated to OVN networks in this project. Format: `UPLINK:192.0.2.0/24`. |
|
restricted.networks.zones |
block |
Comma-separated zone names |
Restricts which DNS domains the project can create network zones for. |
##### 7.5 Cluster Restrictions
|
Key |
Default |
Values |
Description |
|
restricted.cluster.target |
block |
allow, block |
Controls whether users can target specific cluster members when creating or moving instances. When blocked, the scheduler decides placement automatically. |
|
restricted.cluster.groups |
(empty) |
Comma-separated group names |
Restricts which cluster groups instances can be placed into. Only the listed groups are allowed. |
##### 7.6 Backup & Snapshot Restrictions
|
Key |
Default |
Values |
Description |
|
restricted.backups |
block |
allow, block |
Controls whether project users can create instance and volume backups. |
|
restricted.snapshots |
block |
allow, block |
Controls whether project users can create instance and volume snapshots. |
##### 7.7 ID Mapping Restrictions
|
Key |
Default |
Values |
Description |
|
restricted.idmap.uid |
(empty) |
UID range(s) |
Specifies allowed host UID ranges for `raw.idmap`. Format: single ID (`1000`) or range (`1000-2000`). Provides a middle ground: allows raw idmap in a restricted project but only with specific host UIDs. |
|
restricted.idmap.gid |
(empty) |
GID range(s) |
Same as above but for GIDs. Restricts which host GID ranges can be used in `raw.idmap`. |
### 8. Saving Configuration
After making changes to any section (project info, feature flags, or resource limits), click the **Save Configuration** button at the bottom of the page.
- Only changed values are sent to the server (unchanged settings are not affected).
- Changes take effect immediately upon save.
- The page refreshes automatically to show the updated configuration.
- A success or error message will appear indicating the result.
**Feature flag changes:** Toggling feature flags on a project that already contains instances will fail. Remove all instances first, change the flags, then recreate instances.
### 9. Common Use Cases
#### Multi-Tenant Environment
Give each customer or team their own project with isolated resources:
- Enable all feature flags for full isolation
- Set resource limits (e.g., 10 instances, 32 CPUs, 64 GiB memory, 500 GiB disk)
- Enable restricted=true to prevent privilege escalation
- Use `restricted.cluster.groups` to confine instances to specific hardware
#### Dev / Staging / Production Separation
Create separate projects for each environment:
- Share images and networks across environments (features off) but isolate profiles and volumes (features on)
- Set lower resource limits on dev projects, higher on production
- Use restrictions on dev to prevent resource hogging
#### Resource Quota Enforcement
Prevent any single team from consuming all cluster resources:
- Set limits.cpu, limits.memory, and limits.disk per project
- Monitor the Usage % column to see how close each project is to its limits
- Use limits.instances to cap the number of instances
#### Confined User Access
For shared Incus servers where untrusted users manage their own instances:
- Enable restricted=true to block all dangerous features
- Selectively allow specific features (e.g., `restricted.containers.nesting=allow` for Docker users)
- Restrict device access to managed-only (default behavior)
- Restrict TLS client certificates to specific projects
### 10. Best Practices
|
Practice |
Recommendation |
|
**Set feature flags first** |
Configure all feature flags **before** creating any instances in the project. Feature flags are locked once instances exist. |
|
**Always set resource limits in shared environments** |
In multi-project setups, set resource limits on every project. Without limits, one project can consume all cluster resources, starving others. |
|
**Enable restrictions for untrusted users** |
If project users are not fully trusted administrators, enable restricted=true. This prevents privilege escalation, raw configuration access, and device passthrough by default. |
|
**Use descriptions** |
Document each project with a clear description. This helps administrators understand what the project is for, especially in environments with many projects. |
|
**Monitor usage percentages** |
Regularly check the Usage % column in Resource Limits. Projects approaching their limits may need quota increases or instance cleanup. |
|
**Configure the default profile when isolating profiles** |
When features.profiles is enabled, immediately configure the project's own `default` profile with a root disk and network device. Otherwise instance creation will fail. |
|
**Per-pool disk limits for storage tiers** |
In clusters with different storage tiers (HDD, SSD, NVMe), use limits.disk.pool.POOL_NAME via CLI to control how much of each tier a project can consume. |
|
**Restrict cluster targeting in production** |
Keep restricted.cluster.target=block to let the scheduler handle instance placement. This ensures proper load distribution across the cluster. |
---
## Incus – Instances
*Source: `/static/help_incus_instances.html`*
AETHER INCUS Instances Guide
## AETHER INCUS Instances Guide
This guide explains everything about the Instances tab — from understanding instance types and lifecycle states, to performing actions like migration and snapshots. It is designed both as a reference for the AETHER UI and as a study guide for beginning Incus administrators.
**Table of Contents**
- What Are Instances?
- Containers vs Virtual Machines
- Instance Lifecycle and States
- The Instances Toolbar
- Instances Table Columns
- Instance Detail View
- Instance Actions
- Start
- Stop
- Force Stop
- Restart
- Freeze
- Unfreeze
- Bulk Actions
- Migration and Moving Instances
- Migrate to a Different Cluster Member
- Move Root Storage to a Different Pool
- Move to a Different Project
- Snapshots
- Deleting Instances
- Understanding Projects
- Best Practices
### What Are Instances?
In Incus, an **instance** is a running workload — either a **system container** or a **virtual machine (VM)**. Instances are the core building blocks of your infrastructure. They are created from images, configured via profiles and direct settings, and can be managed individually or in bulk.
Each instance has:
- A **unique name** (1–63 characters, lowercase letters, numbers, and dashes)
- A **type** (container or virtual machine)
- A **root disk** stored on a storage pool
- One or more **profiles** that define its default configuration
- A **location** (in clusters: which cluster member it runs on)
- A **project** it belongs to (default: `default`)
### Containers vs Virtual Machines
Incus supports two fundamentally different instance types. Choosing the right type for your workload is one of the most important decisions you will make.
#### System Containers
System containers run a full Linux distribution but **share the host's kernel**. They use Linux kernel features — namespaces (PID, network, mount, user, etc.) and cgroups — to provide isolation. This is software-level isolation, not hardware-level.
- **Performance:** Near-native. No virtualisation overhead. A container can start in under a second.
- **Resource usage:** Minimal. Containers share the host kernel and many system libraries. You can run dozens or hundreds of containers on a single host.
- **OS support:** Linux only. The container must be compatible with the host kernel version.
- **Security:** Isolated via namespaces, AppArmor, seccomp filters, and capability restrictions. By default, containers run unprivileged (mapped to non-root UIDs on the host).
- **Best for:** Linux workloads, microservices, development environments, high-density deployments, CI/CD pipelines.
#### Virtual Machines
Virtual machines are fully virtualised systems powered by **QEMU/KVM**. Each VM runs its own dedicated kernel, providing hardware-level isolation.
- **Performance:** Slight overhead compared to containers due to hardware virtualisation, but modern KVM performance is very good.
- **Resource usage:** Higher. Each VM has its own kernel, init system, and memory space.
- **OS support:** Any operating system, including Windows, BSD, and different Linux distributions with custom kernels.
- **Security:** Stronger isolation boundary. The VM cannot see or interact with the host kernel directly.
- **Special features:** PCI device passthrough (GPUs, network cards), vTPM, UEFI secure boot, VGA/SPICE graphical console.
- **Best for:** Non-Linux operating systems, workloads requiring custom kernels, strong security boundaries, GPU passthrough, desktop environments.
#### Feature Comparison
|
Feature |
System Container |
Virtual Machine |
|
**Kernel** |
Shared with host |
Dedicated per VM |
|
**Isolation** |
Namespaces + cgroups (software) |
Hardware virtualisation (KVM) |
|
**Startup time** |
Sub-second |
Seconds to a minute (full OS boot) |
|
**OS support** |
Linux only |
Any (Linux, Windows, BSD, etc.) |
|
**Resource overhead** |
Minimal |
Higher (dedicated kernel + memory) |
|
**Device passthrough** |
Limited |
Full PCI passthrough (GPUs, NICs) |
|
**Graphical console** |
No (text only) |
Yes (VGA/SPICE) |
|
**Live migration** |
Limited (via CRIU) |
Full support (memory + CPU state) |
|
**Density** |
High (100s per host) |
Lower (10s per host, depending on resources) |
**Tip:** If your workload runs on Linux and does not need a custom kernel, PCI passthrough, or Windows — choose a container. You will get better performance and higher density. Use VMs when you need the stronger isolation boundary or OS flexibility they provide.
### Instance Lifecycle and States
Every instance is always in one of the following states. Understanding these states is fundamental to managing your infrastructure.
|
State |
Colour |
Meaning |
|
Running |
Green |
The instance is active and its processes are executing. For containers, the init process (typically systemd) is running inside the namespace. For VMs, the QEMU process is running and the guest OS is booted. This is the normal operating state for production workloads. |
|
Stopped |
Grey |
The instance is not running. No processes are active and no CPU or memory resources are consumed (beyond the storage space for its root disk). This is the initial state after creating an instance without starting it. Configuration changes can be safely applied in this state. |
|
Frozen |
Blue |
The instance's execution is **suspended** but its state is preserved in memory. For containers, Incus uses the Linux cgroup freezer to halt all processes — they are paused invisibly and resume exactly where they left off when unfrozen. For VMs, the virtual CPU is paused via QEMU. The instance still occupies memory but performs no computation. |
|
Error |
Red |
The instance has entered a failed state and requires intervention. Common causes include image problems, missing prerequisites, or server configuration issues. Use the instance logs (`incus info <name> --show-log`) to diagnose the problem. |
#### State Transitions
Instance State Transitions
Start
+----------+ -----------> +-----------+
| | | |
| Stopped | | Running |<--------+
| | <----------- | | |
+----------+ Stop / +-----------+ Unfreeze
^ Force Stop | | |
| | | |
| Freeze Restart +----------+
| | | | |
| v +------>| Frozen |
| +----------+ | |
| | | +----------+
| | Frozen |
+--- Error <--- +----------+
(any transition
can fail)
**Study note:** Freezing is different from stopping. A stopped instance releases all memory and CPU resources. A frozen instance keeps its memory allocated and processes intact — they are simply paused. Unfreezing is nearly instantaneous, while starting a stopped instance requires a full boot sequence.
### The Instances Toolbar
The toolbar at the top of the Instances tab provides filtering and refresh controls:
|
Control |
Description |
|
**Project Dropdown** |
Select which project's instances to display. Defaults to `default`. When you switch projects, the table reloads with instances from that project. All actions you perform will target instances in the selected project. See Understanding Projects for more details. |
|
**Type Filter** |
Filter instances by type: *All Types*, *Containers*, or *Virtual Machines*. This is a client-side filter — all instances are loaded, and the table is filtered in the browser. |
|
**Status Filter** |
Filter instances by state: *All Statuses*, *Running*, *Stopped*, *Frozen*, or *Error*. Like the type filter, this is applied client-side. |
|
**Search Box** |
Free-text search that filters instances by name. Type any part of an instance name to narrow the list. The filter applies as you type. |
|
**Refresh** |
Reloads the instances list from the Incus cluster. All active filters (project, type, status, search text) are preserved after the refresh. The page also auto-refreshes after you perform an action (start, stop, etc.) on an instance. |
**Tip:** Filters stack. You can combine them — for example, select project `production`, type `Virtual Machines`, status `Running`, and search for `web` to see only running production VMs with "web" in their name.
### Instances Table Columns
The instances table displays the following information. All columns are sortable by clicking on the column header.
|
Column |
Description |
|
**Checkbox** |
Select one or more instances to perform bulk actions. Use the header checkbox to select or deselect all visible instances. |
|
**Name** |
The unique name of the instance. This is a clickable link that opens the Instance Detail View. Instance names follow strict rules: 1–63 characters, only lowercase ASCII letters, digits, and hyphens. Names cannot start with a digit or hyphen, and cannot end with a hyphen. The name is also used as the DNS hostname on the instance's network. |
|
**Type** |
Either `Container` or `Virtual Machine`. See Containers vs Virtual Machines for the differences. |
|
**Status** |
The current state of the instance (Running, Stopped, Frozen, or Error) with colour-coded badges. See Instance Lifecycle and States. |
|
**IPv4** |
The primary IPv4 address of the instance, if available. This is retrieved from the instance's network state. Only shown for running instances with a network interface. |
|
**IPv6** |
The primary IPv6 address of the instance, if available. Similar to IPv4, retrieved from network state. |
|
**Location** |
The cluster member (server) where this instance runs. In a cluster, instances are distributed across members. This column tells you which physical server hosts the instance. Relevant for migration and maintenance planning. |
|
**Snapshots** |
The number of snapshots that exist for this instance. See Snapshots for details. |
|
**Actions** |
Action buttons available for the instance. The available actions depend on the instance's current state. See Instance Actions for a full description of each action. |
### Instance Detail View
Click on an instance name to open its detail view. This shows comprehensive information about the instance including:
- **Status** and **Type**
- **Architecture** (e.g., `x86_64`, `aarch64`)
- **Location** (cluster member)
- **Profiles** applied to the instance
- **Created** date
- **Snapshot count**
- **Network interfaces** with MAC addresses and IP addresses (IPv4 and IPv6)
- **Resource usage** (CPU, memory, disk, network I/O) for running instances
### Instance Actions
Each instance has a set of action buttons in the Actions column. The available actions depend on the instance's current state.
#### Start
Starts a stopped instance. This initiates the full boot sequence:
- The instance's storage volume is mounted from the storage pool.
- Devices (network interfaces, disks, etc.) are initialised.
- For **containers**: the init process (typically `systemd` or `/sbin/init`) is launched inside the container's namespace. The container's AppArmor profile and seccomp filters are applied.
- For **VMs**: QEMU is launched with the configured resources (CPU, memory, disks, devices). The guest OS boots from the virtual disk.
**Available when:** Instance is in **Stopped** state.
#### Stop
Performs a **graceful shutdown** of the instance. This is equivalent to pressing the power button on a physical server:
- For **containers**: a shutdown signal is sent to the init process, which then performs its normal shutdown sequence (stopping services, unmounting filesystems, etc.).
- For **VMs**: an ACPI power-off event is sent to the guest OS, which triggers its shutdown procedure.
Incus waits for the instance to shut down within a configurable timeout (`boot.host_shutdown_timeout`, default: 5 minutes). If the instance does not shut down within this time, it is force-stopped.
**Available when:** Instance is in **Running** state.
#### Force Stop
Immediately **kills** the instance without waiting for a graceful shutdown. This is equivalent to pulling the power cord:
- For **containers**: the container process is terminated immediately.
- For **VMs**: the QEMU process is killed immediately.
**Caution:** Force stop does not give applications inside the instance any time to shut down cleanly. This can cause data loss or corruption — for example, databases with uncommitted transactions, applications with data in write buffers, or filesystems with pending writes. Only use force stop when the instance is unresponsive to a normal stop, or when you need immediate termination.
**Available when:** Instance is in **Running** state.
#### Restart
Performs a graceful **stop followed by a start**. The instance goes through its full shutdown sequence, and once fully stopped, it is started again. This is useful for applying configuration changes that require a restart, or for clearing transient issues.
**Available when:** Instance is in **Running** state.
#### Freeze
**Suspends** the instance while preserving its state in memory:
- For **containers**: Incus uses the Linux **cgroup freezer** to halt all processes in the container's cgroup hierarchy. The processes are frozen in place — they are completely unaware that they have been paused. When unfrozen, they continue exactly where they left off.
- For **VMs**: Incus sends a `stop` command to QEMU via the QMP (QEMU Machine Protocol), which pauses the virtual CPUs. The VM's memory remains allocated.
Freeze is useful for:
- Temporarily freeing CPU resources while preserving memory state
- Taking consistent snapshots of a running workload
- Pausing batch jobs to prioritise interactive workloads
**Available when:** Instance is in **Running** state.
#### Unfreeze
**Resumes** a frozen instance. All processes continue executing from exactly where they were paused. This is nearly instantaneous — there is no boot sequence.
- For **containers**: the cgroup freezer thaws all processes.
- For **VMs**: QEMU resumes the virtual CPUs via a `cont` command.
**Available when:** Instance is in **Frozen** state.
#### Action Availability Summary
|
Current State |
Available Actions |
|
**Running** |
Stop, Force Stop, Restart, Freeze, Migrate, Snapshot, Delete |
|
**Stopped** |
Start, Migrate, Snapshot, Delete |
|
**Frozen** |
Unfreeze, Stop, Force Stop, Snapshot, Delete |
|
**Error** |
Stop, Force Stop, Delete |
### Bulk Actions
When you select one or more instances using the checkboxes, a **bulk action bar** appears at the top of the table. This allows you to perform the same action on multiple instances simultaneously:
- **Start** — Start all selected stopped instances.
- **Stop** — Gracefully stop all selected running instances.
- **Force Stop** — Force stop all selected running instances.
- **Restart** — Restart all selected running instances.
- **Freeze** — Freeze all selected running instances.
- **Unfreeze** — Unfreeze all selected frozen instances.
- **Snapshot** — Create a snapshot for each selected instance.
- **Delete** — Delete all selected instances (with confirmation).
**Note:** Each action is sent to each selected instance individually. Actions that do not apply to a particular instance's current state (e.g., trying to stop an already-stopped instance) will be skipped or return an error for that instance.
**Tip:** Use the **Cluster Members → Evacuate** function to migrate all instances off a cluster member for maintenance. Evacuate is purpose-built for moving all instances off a server, supports multiple migration modes, and handles the process much more gracefully than selecting instances individually.
### Migration and Moving Instances
The **Migrate** button on an instance opens a modal with three options. Each option serves a different purpose:
#### 1. Migrate Instance to a Different Cluster Member
Moves the instance from its current cluster member (server) to a different member in the same cluster.
**When to use:**
- Manual load balancing — move instances away from an overloaded server
- Pre-maintenance preparation — move specific instances before taking a server offline
- Hardware affinity — move an instance to a server with specific hardware (e.g., GPUs, faster storage)
**How it works:**
- The instance's storage and configuration are transferred to the target cluster member.
- For **running VMs** with `migration.stateful=true`: Incus performs a **live migration** — the VM's memory is copied incrementally while it continues running, with only a brief pause at the final switchover (typically milliseconds to seconds). This provides near-zero downtime.
- For **running containers**: live migration is limited. Standard practice is to stop the container, migrate, and restart it.
- For **stopped instances**: the disk data is transferred directly with no downtime concerns.
The dropdown shows all cluster members **except** the member the instance currently runs on.
**Study note:** In the Incus API, intra-cluster migration uses `POST /1.0/instances/{name}` with the body `{"name": "...", "migration": true}` and the query parameter `?target=<member>`. This is different from migration between separate Incus servers, which uses source/target remotes.
#### 2. Move Instance Root Storage to a Different Pool
Moves the instance's root disk from its current storage pool to a different pool on the same cluster. The instance itself stays on the same cluster member.
**When to use:**
- Storage tiering — move an instance from fast SSD storage to cheaper HDD storage (or vice versa)
- Pool maintenance — evacuate instances from a storage pool before decommissioning it
- Changing storage backends — for example, moving from local ZFS to shared Ceph storage for high availability
**How it works:**
- The instance's entire root disk (and all its data) is copied to the new storage pool.
- The instance must be **stopped** for this operation.
- Once the copy is complete, the instance is reconfigured to use the new pool.
The dropdown shows all storage pools **except** the pool the instance's root disk is currently on.
**Note:** Moving storage between pools copies all data, which can take significant time for instances with large root disks. The amount of time depends on the size of the data and the speed of the source and target storage.
#### 3. Move Instance to a Different Project
Moves the instance to a different Incus project. See Understanding Projects for what projects are.
**When to use:**
- Organisational changes — move an instance from a development project to a production project
- Multi-tenancy — reassign instances between teams or departments
- Cleanup — consolidate instances from temporary projects
**How it works:**
- The instance and its data are reassigned to the target project.
- The instance retains its name, configuration, and snapshots.
- The instance must be **stopped** for this operation.
The dropdown shows all projects **except** the project the instance currently belongs to.
### Snapshots
A snapshot is a **point-in-time copy** of an instance's disk state. Snapshots allow you to save the current state of an instance and restore it later if something goes wrong — for example, before applying an update, changing a configuration, or testing a risky operation.
#### How Snapshots Work in AETHER
When you click the **Snapshot** button on an instance (or use the bulk snapshot action), AETHER creates a snapshot with the following defaults:
- **Automatic name:** The snapshot is named using a date-stamp format: `snap-YYYYMMDD-HHMMSS` (e.g., `snap-20260215-143022`). This ensures unique, chronologically sortable names without requiring manual input.
- **7-day expiration:** Each snapshot is created with a **7-day expiry**. After 7 days, Incus automatically deletes the snapshot. This prevents forgotten snapshots from accumulating and consuming storage space.
- **Stateless:** Snapshots created through this interface capture **disk data only** (not memory/CPU state). This means restoring a snapshot restores the disk contents, but the instance will need to be started fresh.
**Tip:** You can take a snapshot of both running and stopped instances. For the most consistent snapshot, consider freezing the instance first (to pause all I/O), taking the snapshot, and then unfreezing. This ensures no writes are in-flight during the snapshot. However, for most workloads, snapshotting a running instance is perfectly safe — modern filesystems handle this well.
#### Snapshots and Storage Pools
Snapshots are stored in the **same storage pool** as the instance. Most storage backends (ZFS, Btrfs, LVM, Ceph) use **copy-on-write** (CoW) technology, which means:
- Creating a snapshot is **nearly instant** regardless of instance size.
- The snapshot initially consumes **almost no additional space** — only changes made after the snapshot consume extra storage.
- Over time, as the instance's data diverges from the snapshot, the snapshot's space usage grows.
#### Stateful vs Stateless Snapshots
While AETHER creates stateless snapshots by default, it is useful to understand the difference for your Incus studies:
|
Type |
Captures |
Restore Behaviour |
Support |
|
**Stateless** |
Disk data only |
Instance restores to the disk state; needs a fresh start |
All instances |
|
**Stateful** |
Disk data + memory + CPU state |
Instance resumes exactly where it was (like waking from hibernation) |
VMs: full support. Containers: limited (requires CRIU). |
Stateful snapshots require `migration.stateful=true` on the instance configuration.
#### Automated Snapshot Scheduling
Beyond the manual snapshots available here, Incus supports automatic snapshot scheduling via instance or profile configuration:
- `snapshots.schedule` — A cron expression or shorthand (`@hourly`, `@daily`, `@weekly`) defining when snapshots are taken automatically.
- `snapshots.expiry` — How long automatic snapshots are kept (e.g., `7d`, `30d`, `1M`).
- `snapshots.pattern` — A naming template for automatic snapshot names.
These settings can be configured in the instance's profile or directly via the Profiles tab. See the Profiles Help for more details.
### Deleting Instances
The **Delete** button permanently removes an instance and all its snapshots. This action cannot be undone.
#### How Deletion Works
- If the instance is **running**, AETHER will **force-stop** it first, then delete it.
- All **snapshots** associated with the instance are also permanently deleted.
- The instance's storage volume is removed from the storage pool.
- A confirmation dialog is shown before deletion proceeds.
**Warning:** Deletion is permanent. There is no recycle bin or undo. Once deleted, the instance's data, configuration, and all snapshots are gone. Always take a snapshot or backup before deleting an instance if you might need its data later.
#### Protecting Against Accidental Deletion
For critical production instances, you can set `security.protection.delete=true` in the instance configuration. This prevents the instance from being deleted until the protection is explicitly removed. This is configured via the Incus CLI:
```
`incus config set <instance-name> security.protection.delete=true`
```
### Understanding Projects
Projects are Incus's way of organising and isolating resources. Think of a project as a **folder** that contains its own set of instances, images, profiles, and (optionally) networks and storage volumes.
#### Why Projects Exist
- **Multi-tenancy:** Give different teams or customers their own isolated environment on shared infrastructure.
- **Organisation:** Separate development, staging, and production workloads.
- **Resource limits:** Set per-project quotas for instances, CPU, memory, and disk.
- **Security:** Restrict what users can do within their project (no privileged containers, limited device access, etc.).
#### The Default Project
Every Incus installation has a `default` project. When you do not specify a project, all operations target the `default` project. This is the project shown when you first load the Instances tab.
#### What Is Isolated per Project
|
Resource |
Always Isolated? |
Notes |
|
**Instances** |
Yes |
Instances always belong to exactly one project. The same instance name can exist in different projects without conflict. |
|
**Images** |
Configurable |
Controlled by `features.images`. When enabled, each project has its own image set. When disabled, images are shared from the `default` project. |
|
**Profiles** |
Configurable |
Controlled by `features.profiles`. When enabled, each project has its own profiles (a `default` profile is auto-created). When disabled, profiles are shared from the `default` project. |
|
**Networks** |
Configurable |
Controlled by `features.networks`. Usually shared across projects unless you need strict network isolation. |
|
**Storage Volumes** |
Configurable |
Controlled by `features.storage.volumes`. Custom volumes can be isolated per project. |
**Important:** Project feature flags (`features.images`, `features.profiles`, etc.) must be set **before** adding any instances to the project. Once a project contains instances, these settings are locked and cannot be changed.
#### Switching Projects in AETHER
Use the **Project dropdown** in the instances toolbar to switch between projects. When you select a different project:
- The instances table reloads showing only instances from that project.
- All actions (start, stop, migrate, snapshot, delete) target instances in the selected project.
- The migrate “Move to a different project” option shows all other projects except the currently selected one.
### Best Practices
#### Resource Management
- **Always set resource limits** — Configure `limits.cpu` and `limits.memory` on instances (via profiles or direct config) to prevent a single instance from consuming all host resources. Without limits, one runaway instance can starve everything else on the host.
- **Use profiles for common settings** — Rather than configuring each instance individually, create profiles for common workload types (e.g., `small` with 1 CPU + 512MB, `medium` with 2 CPUs + 2GB). This ensures consistency and makes changes easier.
- **Monitor your cluster members** — Use the Cluster Members tab to watch memory and CPU load. If a member consistently runs above 70%, consider migrating some instances to less-loaded members.
#### Naming Conventions
- Use descriptive, consistent naming: `web-prod-01`, `db-staging-02`, `app-dev-03`.
- Include the environment or role in the name for easy identification.
- Remember that instance names are used as DNS hostnames — keep them meaningful.
#### Snapshot Strategy
- **Snapshot before changes** — Always take a snapshot before upgrading software, changing configuration, or performing risky operations inside an instance.
- **Use automated schedules for production** — Configure `snapshots.schedule` and `snapshots.expiry` on production instances (via profiles) for regular automatic snapshots.
- **AETHER's snapshots expire after 7 days** — Snapshots created via the Snapshot button are automatically cleaned up after 7 days. If you need longer retention, use Incus snapshot scheduling with a custom expiry.
- **Monitor snapshot counts** — The Snapshots column in the table helps you track how many snapshots each instance has. Excessive snapshots consume storage and can slow down certain operations.
#### Security
- **Use unprivileged containers** — This is the default and should not be changed without a compelling reason. Privileged containers (`security.privileged=true`) have significant security implications — a container escape in a privileged container gives root access to the host.
- **Protect critical instances** — Set `security.protection.delete=true` on production instances to prevent accidental deletion.
- **Use projects for isolation** — In multi-team environments, use projects with `restricted=true` to limit what users can do (e.g., block privileged containers, restrict device access).
#### Migration Planning
- **For planned maintenance** — Use **Evacuate** (on the Cluster Members tab) rather than manually migrating individual instances. Evacuate handles the entire process and remembers which instances to bring back during restore.
- **For single-instance moves** — Use the Migrate button on the Instances tab. This gives you fine-grained control over where to move a specific instance.
- **Enable live migration for critical VMs** — Set `migration.stateful=true` on VMs that cannot tolerate downtime. This enables live migration with near-zero downtime.
- **Plan storage moves for maintenance windows** — Moving an instance's root storage to a different pool copies all data and can take significant time. Schedule these moves during low-usage periods.
**Further reading:** See the other Help guides available on each tab for detailed information about cluster members, storage pools, profiles, and other Incus infrastructure topics.
---
## Deployhelp
*Source: `/static/deployhelp.html`*
New Deployment - Help Guide
## New Deployment - Help Guide
This guide explains how to use the **New deployment** page to deploy virtual machines, containers, and blueprints on your INCUS clusters through the SDN Controller web UI.
### Table of Contents
- Overview
- Understanding Virtual Machines vs Containers
- Understanding Blueprints
- Step 1: Select a Target Cluster
- Step 2: Choose a Deploy Type
- Deploying a Virtual Machine
- Deploying a Container
- Deploying a Blueprint
- Form Fields Reference
- Instance Tags
- Ansible Automation Jobs
- Tips and Best Practices
- Troubleshooting
### Overview
The New deployment page provides a guided, step-by-step form for deploying workloads onto your INCUS infrastructure. The form progressively reveals sections as you make choices — you start by selecting a cluster and deploy type, and additional options appear as needed.
You can deploy three types of workloads:
- **Virtual Machine** — a full operating system with its own kernel, running in hardware-level isolation
- **Container** — a lightweight, fast-starting instance that shares the host kernel
- **Blueprint** — a pre-defined template that deploys one or more instances (VMs and/or containers) as a coordinated group
### Understanding Virtual Machines vs Containers
Choosing between a virtual machine and a container depends on your workload requirements. Both run on the same INCUS cluster, but they differ in important ways:
|
Characteristic |
Virtual Machine |
Container |
|
**Isolation** |
Full hardware-level isolation with its own kernel. Each VM runs a complete operating system independently. |
Process-level isolation sharing the host kernel. Containers are isolated via namespaces and cgroups. |
|
**Startup Time** |
Slower — the VM must boot a full OS kernel (typically seconds to a minute). |
Very fast — starts almost instantly since there is no kernel to boot. |
|
**Resource Overhead** |
Higher — each VM reserves dedicated memory and runs its own kernel processes. |
Lower — containers share the host kernel and have minimal overhead. |
|
**OS Support** |
Can run any operating system (Linux, Windows, BSD, etc.) that has an image available. |
Linux only — the container must be compatible with the host kernel. |
|
**Security** |
Stronger isolation boundary — suitable for untrusted workloads or multi-tenant environments. |
Good isolation for trusted workloads. Not recommended for running untrusted code. |
|
**Storage** |
Requires a root disk image. Disk size is allocated at creation time. |
Uses storage from the pool more efficiently. Root disk size is also set at creation. |
|
**Use Cases** |
Running Windows, legacy applications, applications requiring specific kernels, high-security workloads, or full OS environments. |
Microservices, web servers, development environments, CI/CD runners, or any Linux workload where fast startup and density matter. |
**Tip:** If you are unsure which to pick:
- Choose **Virtual Machine** if you need Windows support, full kernel control, or strong isolation.
- Choose **Container** if you are running Linux workloads and want fast startup with lower resource usage.
### Understanding Blueprints
A **blueprint** is a pre-configured deployment template that defines one or more instances (virtual machines and/or containers) along with their resource allocations, networking, and configuration. Blueprints allow you to deploy complex multi-instance environments in a single operation.
#### When to Use a Blueprint
- **Multi-tier applications** — deploy a web server, application server, and database together as a coordinated stack.
- **Standardized environments** — ensure consistent resource allocation and configuration across deployments.
- **Repeatable deployments** — quickly spin up identical environments for development, testing, or production.
- **Team standards** — administrators can create blueprints that enforce organizational policies for resource sizing, naming conventions, and tagging.
#### How Blueprints Work
- Select "Blueprint" as the deploy type.
- Choose a blueprint from the dropdown — a preview of its components (instances, resources, images) will appear.
- Enter a **base name** — each instance in the blueprint will be named using the pattern `{base}-{component}-{N}`.
- Select a network — all instances in the blueprint will be placed on the same OVN network and receive IPs automatically.
- Review the generated instance names and resource totals, then deploy.
**Note:** When deploying a blueprint, the resource settings (CPU, memory, disk) are defined within the blueprint itself. You do not configure these individually — they are shown in the blueprint component preview so you can review them before deploying.
### Step 1: Select a Target Cluster
The first step is to select the INCUS cluster where your instance(s) will be deployed.
- The **Cluster** dropdown lists all clusters you have access to.
- Once you select a cluster, the system loads the available images, networks, storage pools, and resource information for that cluster.
- If you only have access to one cluster, it may be pre-selected for you.
**Note:** The available images, networks, and storage pools depend on the cluster you select. Different clusters may have different resources available.
### Step 2: Choose a Deploy Type
After selecting a cluster, the **Deploy Type** section appears with three options:
- **Virtual Machine** — deploys a single VM.
- **Container** — deploys a single container.
- **Blueprint** — deploys one or more instances from a pre-configured template.
Selecting a deploy type reveals the appropriate form fields for that type. See the sections below for a walkthrough of each.
### Deploying a Virtual Machine
Follow these steps to deploy a virtual machine:
- **Select an Image** — Choose the operating system image for the VM. The dropdown shows all images available on the selected cluster. The image description and type are shown to help you identify the correct one.
- **Enter an Instance Name** — Provide a unique name for the VM. The name must:
- Start with a letter (a-z or A-Z)
- Contain only letters, numbers, and dashes
- Be between 1 and 63 characters long
A green checkmark appears when the name is valid and available on the cluster. A red X indicates a conflict or invalid format.
- **Network Configuration** — Select an OVN network and an IP address:
- The **OVN Network** dropdown lists all available networks on the cluster.
- After selecting a network, the **IP Address** dropdown shows available IPs within that network's subnet. The system automatically excludes IPs that are already in use.
- **Resources** — Set the CPU and memory allocation:
- **CPU**: Choose from 1 to 64 CPUs.
- **Memory**: Enter the amount in GiB or MiB. The cluster's total and available memory is shown for reference.
- **Storage** — Configure the disk:
- **Root Disk Size**: Enter the size in GiB. This is the primary disk of the VM.
- **Storage Pool**: Select which storage pool to use. Available space is shown for each pool.
- **Additional Disks** (optional): Click "+ Add Disk" to attach extra disks. Each additional disk requires a name, size, and storage pool.
- **Instance Tags** (optional) — Add key-value metadata tags. Tags are useful for organizing instances and for use in firewall rules (ACLs). See Instance Tags below.
- **Deploy** — Click the "Deploy Instance" button. A confirmation dialog may appear. The deployment progress is shown in real time.
**Important:** Make sure the cluster has sufficient resources (CPU, memory, storage) before deploying. The form shows available resources to help you plan, but over-committing resources can affect performance of existing workloads.
### Deploying a Container
Deploying a container follows the same steps as a virtual machine, with a few differences:
- **Select an Image** — The image list is filtered to show container-compatible images (Linux only).
- **Enter an Instance Name** — Same naming rules as VMs.
- **Network Configuration** — Same as VMs — select an OVN network and IP address.
- **Resources** — Same CPU and memory options. Containers typically need fewer resources than VMs for equivalent workloads.
- **Storage** — Same root disk and storage pool options. Containers generally use less disk space than VMs.
- **Instance Tags** (optional) — Same as VMs.
- **Deploy** — Click "Deploy Instance".
**Tip:** Containers start much faster than VMs. If you are deploying a Linux workload and do not require full kernel isolation, containers are a more efficient choice.
### Deploying a Blueprint
Deploying a blueprint creates multiple instances in a single operation:
- **Select a Blueprint** — The dropdown lists all available blueprints. After selecting one, a **component preview** appears showing:
- Each instance that will be created (name pattern, type, image)
- Resource allocation per instance (CPU, memory, disk)
- Total resource requirements for the entire blueprint
- **Enter a Base Name** — This name is used as a prefix for all instances. For example, if your base name is `my-app` and the blueprint has components named "web" and "db", the instances will be named `my-app-web-1` and `my-app-db-1`. The generated names are previewed in real time as you type.
- Must start with a letter
- Only letters, numbers, and dashes allowed
- Maximum 41 characters
- **Network Configuration** — Select an OVN network. All instances in the blueprint will be placed on this network and receive IP addresses automatically.
- **Deploy** — Click "Deploy Blueprint". A progress modal appears showing the deployment status of each instance. You can:
- **Cancel** — stop the deployment (instances already created will remain)
- **Run in Background** — continue the deployment while you navigate away from the page
**Note:** Blueprint deployments create instances sequentially. If one instance fails to deploy, the process stops and reports the error. Successfully deployed instances from the same blueprint remain running and are not rolled back.
### Form Fields Reference
Quick reference for all form fields and their validation rules:
|
Field |
Required |
Rules |
|
**Cluster** |
Yes |
Must select a cluster from the dropdown. |
|
**Deploy Type** |
Yes |
Select one: Virtual Machine, Container, or Blueprint. |
|
**Blueprint** |
Yes (blueprint only) |
Select a blueprint from the dropdown. |
|
**Base Name** |
Yes (blueprint only) |
Start with a letter, letters/numbers/dashes only, max 41 characters. |
|
**Image** |
Yes (VM/Container) |
Select an image from the dropdown. Available images depend on the cluster and deploy type. |
|
**Instance Name** |
Yes (VM/Container) |
Start with a letter, letters/numbers/dashes only, 1-63 characters. Must be unique on the cluster. |
|
**OVN Network** |
Yes |
Select a network. Only OVN networks managed by the SDN Controller are listed. |
|
**IP Address** |
Yes (VM/Container) |
Select an available IP. IPs already assigned to other instances are excluded. |
|
**CPU** |
Yes (VM/Container) |
Select from 1, 2, 4, 8, 16, 32, or 64 CPUs. |
|
**Memory** |
Yes (VM/Container) |
Enter a value in GiB or MiB. Minimum 1, maximum 1024. |
|
**Root Disk Size** |
Yes (VM/Container) |
Size in GiB. Minimum 1, maximum 10000. |
|
**Storage Pool** |
Yes (VM/Container) |
Select a storage pool. Available space is shown. |
|
**Additional Disks** |
No |
Optional extra disks. Each needs a name, size (GiB), and storage pool. |
|
**Instance Tags** |
No |
Key-value pairs. Max 10 tags. Keys may contain letters, numbers, dots, dashes, and underscores. |
### Instance Tags
Instance tags are key-value metadata pairs stored on the instance in INCUS (as `user.{key}` configuration). They serve two main purposes:
- **Organization** — categorize and identify instances (e.g., `department=finance`, `environment=production`).
- **Firewall Rules** — use tags in SDN Controller ACL rules to dynamically match instances. For example, a firewall rule with source `env=prod` automatically applies to all instances tagged with `env=prod`.
To add a tag:
- Enter a **Key** (e.g., `environment`)
- Enter a **Value** (e.g., `production`)
- Click the **+** button to add it
You can add up to 10 tags per instance. Tags can be modified after deployment through the instance management page.
**Tip:** Plan your tagging strategy before deploying. Consistent tag naming across instances makes it much easier to create effective firewall rules later. Common tag keys include: `env`, `role`, `app`, `tier`, `department`, and `owner`.
### Ansible Automation Jobs
At the bottom of the deployment page, a table shows **Your Recent Ansible Automation Jobs**. This displays the history of Ansible/AWX jobs that were triggered as part of your deployments.
If a blueprint or deployment configuration includes post-deploy Ansible automation (e.g., running a configuration playbook after an instance is created), the job status and results appear here. The table shows:
- **Time** — when the job was triggered
- **Instance** — which instance the job relates to
- **Cluster** — which cluster the instance is on
- **Event** — what triggered the job
- **AWX Job ID** — the job ID in AWX for further investigation
- **Status** — current job status (running, successful, failed)
- **Result** — job outcome details
Use the search box to filter jobs by any column value.
### Tips and Best Practices
#### Choosing the Right Resources
- **CPU**: Start with fewer CPUs (1-2) for lightweight workloads. You can increase later if needed. Over-allocating CPU wastes cluster capacity.
- **Memory**: Check the application's requirements. Most Linux containers run well with 1-2 GiB. VMs typically need 2-4 GiB minimum depending on the OS.
- **Disk**: Size the root disk for the OS plus application data. Use additional disks for data that may grow independently. It is easier to add storage later than to shrink it.
#### Naming Conventions
- Use descriptive names that identify the purpose: `web-prod-01`, `db-staging-02`, `app-dev-frontend`.
- Include the environment in the name or tags for easy identification.
- For blueprints, choose a base name that describes the application stack: `ecommerce`, `monitoring`, `crm-staging`.
#### Network Planning
- Place instances that need to communicate frequently on the same OVN network.
- Use different networks to isolate environments (production, staging, development).
- The IP addresses shown in the dropdown are guaranteed to be available at the time of loading. If another user deploys at the same time, a conflict will be reported and you can retry with a different IP.
#### Storage Pool Selection
- Check the available space shown next to each pool before selecting.
- Distribute workloads across pools to avoid filling a single pool.
- Different pools may use different storage drivers (ZFS, LVM, etc.) with different performance characteristics.
### Troubleshooting
|
Problem |
Possible Cause |
Solution |
|
No clusters in dropdown |
Your user account does not have access to any clusters. |
Contact your administrator to grant cluster access. |
|
No images available |
The selected cluster has no images imported, or no images match the selected deploy type. |
Ask an administrator to import images to the cluster. |
|
Instance name shows red X |
The name is already in use on the cluster, or does not meet the naming rules. |
Choose a different name. Names must start with a letter and contain only letters, numbers, and dashes. |
|
No available IP addresses |
All IPs in the selected network's subnet are assigned. |
Select a different network, or free up IPs by removing unused instances. |
|
Deployment fails with resource error |
The cluster does not have enough CPU, memory, or storage for the requested instance. |
Reduce the resource allocation, choose a different cluster, or free up resources. |
|
Blueprint deploy partially completes |
One instance in the blueprint failed to deploy. |
Check the error message in the progress modal. Successfully deployed instances remain running. Fix the issue and deploy the remaining instances manually or retry. |
|
No blueprints in dropdown |
No blueprints have been created, or none are available for the selected cluster. |
Create a blueprint in the Blueprints management page, or ask an administrator. |
|
Storage pool shows 0 available |
The storage pool is full. |
Select a different storage pool or free up space by removing unused volumes or instances. |
**Note:** If a deployment fails, check the error message carefully. The SDN Controller provides specific error messages from INCUS that indicate exactly what went wrong. Common issues include name conflicts, insufficient resources, and network configuration problems.
---
## Healthhelp
*Source: `/static/healthhelp.html`*
AETHER Health Page Guide
## AETHER Infrastructure Health Page Guide
This guide explains how to interpret the values displayed on the AETHER Infrastructure Health page. The health page provides a real-time overview of your INCUS clusters, HAProxy load balancers, and storage pools.
### Summary Cards (Top Row)
The five summary cards at the top provide a quick overview of your infrastructure health:
|
Card |
Format |
Description |
|
**CLUSTERS** |
`X/Y` |
X = clusters with all members online, Y = total clusters |
|
**HAPROXY INFRA** |
`X/Y` |
X = HAProxy pairs with both instances running, Y = total pairs |
|
**HAPROXY SERVICES** |
`X/Y` |
X = services with at least 1 backend responding, Y = total services (clickable - see below) |
|
**STORAGE** |
`X/Y` |
X = storage pools with status OK and usage below 85%, Y = total pools |
|
**ANSIBLE** |
`X/Y` |
X = Ansible/AWX endpoints that are reachable, Y = total endpoints |
#### Card Color Coding
|
Color |
Meaning |
Condition |
|
Green |
All healthy |
X equals Y (100% healthy) |
|
Orange |
Some issues |
X is greater than half of Y |
|
Red |
Critical |
X is less than or equal to half of Y |
### HAProxy Services Card
The **HAPROXY SERVICES** summary card (one of the five cards at the top of the page) shows a summary of all HAProxy services:
- **Format:** `X/Y`
- **X** = Services that are operational (have at least 1 backend responding)
- **Y** = Total number of services across all HAProxy infrastructures
**Tip:** This card is clickable! Click on the HAPROXY SERVICES card to open a detailed pop-up modal showing each service with its backends and their individual health status.
#### Services Modal Columns
|
Column |
Description |
|
**Service** |
Name of the HAProxy service |
|
**VIP:Port** |
The service VIP address and listen port (e.g., `10.0.0.50:443`) |
|
**LB VIP** |
The OVN Load Balancer VIP address with port 443 (e.g., `192.168.1.100:443`) |
|
**Backends** |
Individual backend servers with their health status badges |
|
**Status** |
Overall service status showing backends up/total (e.g., `3/4`) |
#### LB VIP Color Coding
The LB VIP column shows the OVN Load Balancer VIP reachability:
|
Color |
Meaning |
|
Green |
LB VIP is reachable (TCP connect to port 443 successful) |
|
Red |
LB VIP is unreachable (TCP connect to port 443 failed) |
**Note:** The LB VIP reachability test is performed from one of the running HAProxy instances by attempting a TCP connection to port 443 on the OVN Load Balancer VIP.
### INCUS Clusters Table
The clusters table shows detailed information for each INCUS cluster:
|
Column |
Description |
|
**Cluster** |
Name of the INCUS cluster (sorted alphabetically) |
|
**Members** |
Online members / Total members in the cluster |
|
**CPU** |
Total CPU cores across all cluster members (aggregated) |
|
**Memory** |
Used / Total memory across all cluster members with usage bar |
|
**HAProxy** |
HAProxy load balancers deployed on this cluster (see below) |
|
**Status** |
Cluster health status: Healthy, Degraded, or Down |
#### Members Badge
|
Color |
Meaning |
|
Green |
All members online (e.g., `8/8`) |
|
Orange |
Some members offline but cluster still operational (e.g., `7/8`) |
|
Red |
All members offline (e.g., `0/8`) |
**Tip:** The members badge is clickable! Click on it to open a detailed pop-up modal showing each cluster member with its status, roles, and current message.
#### Members Modal
When you click on the members badge, a modal opens showing detailed information about each cluster member:
|
Column |
Description |
|
**Server** |
Name of the cluster member server |
|
**Status** |
Current status of the member (Online, Offline, Evacuated, Blocked) |
|
**Roles** |
Cluster roles assigned to this member (database, database-leader, stand-by) |
|
**Message** |
Current status message from INCUS (e.g., "Fully operational", "No heartbeat since...") |
##### Member Status Colors
|
Status |
Color |
Meaning |
|
Online |
Green |
Member is healthy and responding to heartbeats |
|
Offline |
Red |
Member is unreachable (no heartbeat received) |
|
Evacuated |
Yellow |
Member has been intentionally drained of workloads |
|
Blocked |
Gray |
Member is blocked from receiving new workloads |
##### Member Roles
|
Role |
Description |
|
**database-leader** |
This member is the current leader of the distributed database (highlighted in blue) |
|
**database** |
This member participates in the distributed database (voter) |
|
**stand-by** |
This member does not participate in the database (non-voter, runs workloads only) |
**Note:** In INCUS clusters, typically 3 members have the database role for quorum. Additional members are stand-by nodes that run workloads but don't vote on database writes. If a database member fails, it is NOT automatically replaced by a stand-by member - this requires manual intervention.
#### Cluster Status
|
Status |
Meaning |
|
Healthy |
All cluster members are online |
|
Degraded |
Some members are offline but cluster is still operational |
|
Down |
No members are online or cluster is unreachable |
### HAProxy Column (In Clusters Table)
Each HAProxy load balancer deployed on a cluster is displayed as a compact badge:
```
`shortname [instance-dots] X/Y services, A/B backends`
```
#### Components
|
Component |
Description |
|
**shortname** |
Shortened HAProxy pair name (e.g., `ffsdn-haproxy`) |
|
**Instance dots** |
Two dots showing instance status (see below) |
|
**X/Y services** |
X = services with at least 1 backend up, Y = total services |
|
**A/B backends** |
A = backends responding (UP), B = total backends |
#### Status Indicators (Dots)
Three indicators show the status of the HAProxy pair:
|
Indicator |
Description |
Bright = OK |
Dark = Problem |
|
**Dot 1** |
HAProxy instance 01 status |
Instance running |
Instance stopped/unreachable |
|
**Dot 2** |
HAProxy instance 02 status |
Instance running |
Instance stopped/unreachable |
|
**V** |
OVN VIP reachability |
VIP accepts TCP connection on port 443 |
VIP unreachable |
**Note:** The VIP reachability test is performed by a TCP connect to port 443 on the OVN VIP from one of the running HAProxy instances. This confirms the load balancer is accepting connections.
#### HAProxy Badge Color
|
Color |
Meaning |
Condition |
|
Green |
Healthy |
Both instances running AND VIP reachable AND all backends UP |
|
Yellow |
Degraded |
Both instances running AND VIP reachable BUT some backends are down (services still functional) |
|
Orange |
Warning |
One HAProxy instance down (failover active), but VIP still reachable |
|
Red |
Critical |
Both instances down, OR VIP unreachable, OR any service has all backends down |
#### Examples
|
Display |
Interpretation |
|
myapp-haproxy 1/1 service, 4/4 backends |
Healthy: 1 service up, all 4 backends responding |
|
myapp-haproxy 1/1 service, 3/4 backends |
Degraded (yellow): Service is up but 1 backend is not responding |
|
myapp-haproxy 2/2 services, 8/8 backends |
Healthy: 2 services, all 8 backends responding |
|
myapp-haproxy 2/2 services, 7/8 backends |
Degraded (yellow): Both services up, but 1 backend not responding |
|
myapp-haproxy 1/1 service, 4/4 backends |
Warning (orange): One HAProxy instance is down (failover active) |
|
myapp-haproxy 1/2 services, 2/8 backends |
Critical: 1 service has all backends down |
**Note:** A cluster without HAProxy infrastructure will show `-` in the HAProxy column.
### Storage Pools Table
The storage table shows all storage pools across all clusters (sorted by cluster name, then pool name):
|
Column |
Description |
|
**Cluster** |
Name of the INCUS cluster this pool belongs to |
|
**Pool** |
Name of the storage pool |
|
**Driver** |
Storage driver type (e.g., zfs, lvm, btrfs, dir) |
|
**Usage** |
Visual bar showing used/total space with percentage |
|
**Status** |
Pool status from INCUS (typically "Created") |
#### Usage Bar Colors
|
Color |
Meaning |
Threshold |
Navigation Impact |
|
Green |
Healthy |
Below 85% |
No indicator |
|
Yellow |
Degraded |
85% - 89% |
AETHER Health link turns yellow |
|
Orange |
Warning |
90% - 94% |
AETHER Health link turns orange |
|
Red |
Critical |
95% and above |
AETHER Health link turns red |
**Warning:** Storage pools at 85% or above usage require attention. Pools at 95% or above are critical and may cause service disruption if not addressed immediately.
### Memory Usage Bar
Memory usage in the clusters table follows the same color thresholds as storage:
|
Color |
Meaning |
Threshold |
Navigation Impact |
|
Green |
Healthy |
Below 85% |
No indicator |
|
Yellow |
Degraded |
85% - 89% |
AETHER Health link turns yellow |
|
Orange |
Warning |
90% - 94% |
AETHER Health link turns orange |
|
Red |
Critical |
95% and above |
AETHER Health link turns red |
**Warning:** Clusters at 85% memory usage or above may experience performance degradation. Clusters at 95% or above are critical and may cause out-of-memory conditions.
### CPU Load Bar
CPU load in the clusters table shows the 5-minute load average as a percentage of total CPU cores:
|
Color |
Meaning |
Threshold |
Navigation Impact |
|
Green |
Healthy |
Below 70% |
No indicator |
|
Yellow |
Degraded |
70% - 79% |
AETHER Health link turns yellow |
|
Orange |
Warning |
80% - 94% |
AETHER Health link turns orange |
|
Red |
Critical |
95% and above |
AETHER Health link turns red |
**Warning:** Clusters at 70% CPU load or above may experience performance issues. Clusters at 95% or above are critical and may cause service degradation.
### Navigation Health Indicator
The **AETHER Health** link in the sidebar navigation automatically changes color to indicate infrastructure issues. This runs as a background check every 5 minutes (or as configured) and at application startup.
|
Color |
Status |
Triggering Conditions |
|
Normal (no highlight) |
Healthy |
All systems operating normally |
|
Yellow |
Degraded |
- Aether server disk usage 85-89%
|
|
Orange |
Warning |
- Cluster member offline but cluster still operational (e.g., 2/3 members online)
- One HAProxy instance down (failover active)
- Some Ansible/AWX endpoints unreachable
- Some HAProxy service backends down (services still functional)
- Storage pool usage not OK (85-94%)
- CPU load 80-94%
- Memory usage 90-94%
- Aether server disk usage 90-94%
|
|
Red |
Critical |
- All cluster members offline
- Both HAProxy instances down
- VIP unreachable
- LB service completely down (all backends down)
- All Ansible/AWX endpoints down
- CPU load 95% or above
- Memory usage 95% or above
- Storage pool usage 95% or above
|
**Note:** The navigation indicator shows the *worst* condition across all monitored infrastructure. The indicator updates when you navigate to any page or refresh. For real-time updates, visit the Health page directly.
### Ansible Endpoints Table
The Ansible endpoints table shows all configured AWX/Ansible automation endpoints:
|
Column |
Description |
|
**Name** |
Configured name of the Ansible endpoint |
|
**URL** |
Base URL of the AWX/Ansible Tower server (clickable link) |
|
**Version** |
AWX version number if the endpoint is reachable |
|
**Status** |
Endpoint health status |
#### Ansible Endpoint Status
|
Status |
Meaning |
|
Healthy |
Endpoint is reachable and responding to API requests |
|
Down |
Endpoint is unreachable or not responding |
**Note:** Ansible endpoint health is checked by making an authenticated API request to the AWX server. A failed connection or authentication error will show the endpoint as Down.
### Health Event Logging
The health monitoring system automatically logs state changes to the **Web UI Log**. Only actual changes are logged (not every check cycle), providing a clear audit trail of infrastructure events.
#### Log Levels
|
Level |
Color Trigger |
Example Events |
|
**INFO** |
Yellow (degraded) or Recovery (back to green) |
- Backend recovered: DOWN -> UP
- Storage pool usage changed: 80% -> 86%
- LB Service recovered: DOWN -> UP
|
|
**WARN** |
Orange (warning) |
- Ansible endpoint became unreachable
- CPU load entered warning range
- Storage/Memory usage 90-94%
|
|
**ERROR** |
Red (critical) |
- Backend changed: UP -> DOWN
- HAProxy instance changed: Running -> Stopped
- LB Service changed: UP -> DOWN
- Cluster member went offline
- VIP became unreachable
|
#### Log Message Format
Log messages include identifiable information for easy searching:
```
`[Health] Backend web-server-01 (10.0.1.5) in service my-app on cluster Production changed: UP -> DOWN
[Health] LB Service my-app on cluster Production changed: UP -> DOWN
[Health] HAProxy instance prod-haproxy-01 on cluster Production changed: Running -> Stopped
[Health] Storage pool default on cluster Production changed: 84% -> 91%
[Health] Backend web-server-01 (10.0.1.5) in service my-app on cluster Production recovered: DOWN -> UP`
```
**Note:** Health events are logged under the username "SYSTEM". You can filter the Web UI Log by this username to see all automated health events.
### Auto-Refresh
Use the auto-refresh dropdown to automatically update the health data:
- **Off:** Manual refresh only (click the Refresh button)
- **30s / 60s / 2m / 5m:** Automatic refresh at the selected interval
Your preference is saved in your browser and will persist across sessions.
### Data Sources
The health page queries data directly from:
- **INCUS API:** Cluster members (`/1.0/cluster/members`), resources (`/1.0/resources`), storage pools (`/1.0/storage-pools`)
- **HAProxy stats:** Via INCUS exec running `show stat` on the HAProxy admin socket
- **AWX API:** Ansible endpoint health and version information
- **FFSDN Database:** HAProxy infrastructure configuration, service definitions, and AWX endpoint settings
**Note:** CPU and memory values are aggregated totals from all online cluster members. If a member is offline, its resources are not included in the total.
---
## Traceflowhelp
*Source: `/static/traceflowhelp.html`*
Trace Network Flow - Help
## Trace Network Flow - Help
### What is Trace Network Flow?
The Trace Network Flow feature allows you to test and visualize which firewall rules (ACLs) would match a specific network flow between two IP addresses. This is useful for:
- Verifying that traffic between two instances will be allowed or blocked
- Troubleshooting connectivity issues by identifying which ACL rule is being applied
- Testing firewall rules before deploying changes to production
- Understanding how your ACL configuration affects specific traffic patterns
### How to Use the Form
The Trace Network Flow form requires you to specify the details of the network flow you want to trace:
#### Source IP Address
- **Description**: The IP address where the network traffic originates from.
- **Format**: A valid IPv4 address (e.g., `192.168.1.100`).
- **Example**: If you want to test traffic from a web server at `10.105.232.35`, enter that IP here.
#### Destination IP Address
- **Description**: The IP address where the network traffic is going to.
- **Format**: A valid IPv4 address (e.g., `8.8.8.8`).
- **Example**: If you want to test traffic going to a database server at `10.0.5.20`, enter that IP here.
#### Protocol
- **Description**: The network protocol used for the traffic.
- **Options**:
- `TCP` - Transmission Control Protocol, used for most application traffic (HTTP, HTTPS, SSH, databases, etc.)
- `UDP` - User Datagram Protocol, used for DNS, streaming, gaming, etc.
- `ICMP` - Internet Control Message Protocol, used for ping and network diagnostics
**Note:** When you select ICMP, the Port field is automatically disabled because ICMP does not use ports.
#### Port
- **Description**: The destination port number for TCP or UDP traffic.
- **Format**: A number between 1 and 65535.
- **Examples**:
- `80` - HTTP
- `443` - HTTPS
- `22` - SSH
- `5432` - PostgreSQL
- `3306` - MySQL
**Note:** The Port field is only required when using TCP or UDP protocols. It is disabled for ICMP.
### Understanding the Results
After submitting the form, a popup will display the trace results. Here is what each field means:
|
Field |
Description |
|
**INCUS/OVN Cluster Name** |
The name of the INCUS cluster where the source or destination instance resides. |
|
**Source Instance Name** |
The name of the INCUS instance that matches the source IP address (if found). |
|
**Destination Instance Name** |
The name of the INCUS instance that matches the destination IP address (if found). This only appears when both source and destination are managed instances. |
|
**NIC Name** |
The network interface card name on the instance that has the matching IP address. |
|
**OVN Network Name** |
The name of the OVN network that the instance's NIC is connected to. |
|
**Name of ACL that this flow hits** |
The name of the Access Control List (ACL) that contains the matching rule. ACL names follow the format `FFSDNIO-<ruleID>-<direction>` for local rules or `FFSDNIOGLB-<ruleID>-<direction>` for global rules. |
|
**First Matching Rule** |
Details of the first firewall rule that matches the traced flow, including:
- **Direction**: Ingress (incoming) or Egress (outgoing)
- **Action**: ALLOW, DROP, or REJECT
- **Protocol**: TCP, UDP, or ICMPv4
- **Source/Destination**: The IP addresses or ranges in the rule
- **Port**: The port number (or N/A for ICMP)
|
### Example Use Cases
#### Testing Web Server Access
To verify that external clients can reach your web server on port 443:
```
`Source IP: 0.0.0.0 (or any external IP)
Destination IP: 10.105.232.35 (your web server)
Protocol: TCP
Port: 443`
```
#### Testing Database Connectivity
To check if your application server can connect to your PostgreSQL database:
```
`Source IP: 10.0.1.50 (application server)
Destination IP: 10.0.5.20 (database server)
Protocol: TCP
Port: 5432`
```
#### Testing ICMP/Ping
To verify that ping is allowed between two instances:
```
`Source IP: 192.168.1.10
Destination IP: 192.168.1.20
Protocol: ICMP
Port: (disabled)`
```
### Troubleshooting
|
Issue |
Possible Cause |
Solution |
|
Invalid Source/Destination IP address error |
The IP address format is incorrect |
Ensure you enter a valid IPv4 address in the format `xxx.xxx.xxx.xxx` where each octet is between 0 and 255 |
|
Port must be a number between 1 and 65535 |
Invalid port number entered |
Enter a valid port number between 1 and 65535 |
|
No instance found for the IP |
The IP address does not belong to any managed INCUS instance |
Verify the IP address is correct and belongs to an instance in a managed cluster |
|
Default rule shown |
No specific ACL rule matches the flow |
The flow is hitting the network's default action (allow or drop). Create a specific ACL rule if you need different behavior |
**Important:** The Trace Network Flow feature shows which rule *would* match the specified flow based on current ACL configuration. It does not send actual network traffic. Use this tool for planning and troubleshooting, not as a connectivity test.
---
## NewACL
*Source: `/static/newACL.html`*
Creating a New Firewall Rule in AETHER
## Creating a New Firewall Rule in SDN Controller
This guide explains how to fill in the "Create new SDN Firewall Rules for this Cluster" form in the SDN Controller web UI after clicking the "Create" button under "Manage Cluster". Firewall rules define traffic control for an INCUS cluster, applied to instances or OVN networks. These are **local cluster ACLs**, specific to one cluster. For organization-wide rules, see Local Cluster ACLs vs. Global ACLs.
### Prerequisites
- You must be logged into the SDN Controller web UI as a user with access to manage the specific cluster (either `admin` or a user with cluster access).
- Navigate to "Clusters" > "Manage" for your cluster, then click "Create" under "Create new SDN Firewall Rules for this Cluster".
- Basic understanding of network protocols (TCP, UDP, ICMP) and firewall rules.
### Local Cluster ACLs vs. Global ACLs
The SDN Controller supports two types of Access Control Lists (ACLs): **local cluster ACLs** and **global ACLs**. Understanding their differences is crucial for effective firewall rule management.
- **Local Cluster ACLs**:
- **Scope**: Apply only to a specific INCUS cluster (e.g., `MyCluster`).
- **Where Created**: In the "Manage Cluster" page, under "Create new SDN Firewall Rules for this Cluster" (this guide).
- **Storage**: Stored in the `FFSDNIO_rules` database table with a `cluster_id`.
- **Naming**: Named as `FFSDNIO--` (e.g., `FFSDNIO-129-egress`).
- **Use Case**: For cluster-specific policies, such as allowing HTTP traffic only in one cluster's production environment.
- **Access**: Available to `admin` or users with access to the cluster (via "Cluster Access").
- **Global ACLs**:
- **Scope**: Apply to **all INCUS clusters** managed by the SDN Controller.
- **Where Created**: In the "Global ACLs" page (accessible via the navigation menu, admin-only).
- **Storage**: Stored in the `ffsdnioglb_rules` database table, without a `cluster_id`.
- **Naming**: Named as `FFSDNIOGLB--` (e.g., `FFSDNIOGLB-2-egress`).
- **Use Case**: For organization-wide policies, such as allowing DNS traffic from all production instances across all clusters.
- **Access**: Restricted to `admin` users only.
**Note:** Use local cluster ACLs for targeted control within a single cluster. Use global ACLs for consistent rules across all clusters, but ensure they don't conflict with local rules, as both are applied to instances or networks. Check the "Global ACLs" page for existing global rules before creating local ones.
### Step 1: Understand the Form Layout
This form creates **local cluster ACLs** for the selected cluster. For global ACLs, navigate to "Global ACLs" (admin-only). The form includes:
- **Rule Fields** (one or more rows):
- `Direction`: IN/OUT (both), Ingress/In (inbound), or Egress/Out (outbound).
- `Source`: Source IP, CIDR, or instance tag condition.
- `Destination`: Destination IP, CIDR, or instance tag condition.
- `Protocol`: TCP, UDP, or ICMP.
- `Port`: Port numbers or ranges (for TCP/UDP only).
- `Action`: Allow, Drop, or Reject.
- `Description`: A text description of the rule.
- **Buttons**: "Add another FW rule" and "Review firewall rules".
### Step 2: Define Firewall Rules
Each rule row specifies a firewall rule for the current cluster. Start with one row and add more as needed using "Add another FW rule".
#### Direction
- **Options**: `IN/OUT` (both directions), `Ingress/In` (inbound traffic), `Egress/Out` (outbound traffic).
- **Validation**: Must select one (cannot be empty).
- **Example**: `Ingress/In` for incoming web traffic.
#### Source and Destination
- **Format**:
- IPv4 address (e.g., `192.168.1.10`).
- CIDR notation (e.g., `192.168.1.0/24` or `0.0.0.0/0` for ANY).
- Instance tag formula with operators (see Instance Tag Operators below).
- Comma-separated mix of the above (e.g., `0.0.0.0/0,10.34.2.1/24,env=prod OR (app=web AND env=acc),10.6.5.32`).
- **Validation**:
- Cannot be empty (use `0.0.0.0/0` to represent ANY source or destination).
- IPv4: Must match `xxx.xxx.xxx.xxx` where each octet is 0-255.
- CIDR: Must be `xxx.xxx.xxx.xxx/yy` where subnet mask `yy` is 0-32.
- Instance tags: Must contain an operator (`=`, `=^`, or `=*`), and use valid logical operators (`AND`, `OR`) with parentheses if complex.
- Comma-separated: Each part must be a valid IPv4, CIDR, or instance tag formula, separated by commas (no spaces after commas).
- Invalid examples: `256.1.2.3`, `10.0.0.1/33`, `role` (no operator), `10.0.0.1, role=web` (space after comma), empty field.
- **Example**:
```
`Source: 192.168.1.0/24,10.0.3.1,env=prod
Destination: role=webserver,10.6.5.32,(app=web AND env=acc)`
```
```
`Source: 0.0.0.0/0
Destination: role=webserver`
```
**Note:** To allow traffic from or to any IP address, explicitly enter `0.0.0.0/0` in the `Source` or `Destination` field. Leaving these fields empty is not allowed and will prompt a validation error.
#### Instance Tag Operators
Instance tag formulas allow you to dynamically match instances based on their INCUS instance tags. The SDN Controller supports three matching operators:
|
Operator |
Syntax |
Description |
Example |
Matches |
|
**Equals** |
`=` |
Exact match - tag value must equal the specified value exactly |
`env=prod` |
Instances where `env` is exactly `prod` |
|
**Starts With** |
`=^` |
Prefix match - tag value must start with the specified value |
`env=^prod` |
Instances where `env` starts with `prod` (e.g., `prod`, `prod-eu`, `production`) |
|
**Contains** |
`=*` |
Substring match - tag value must contain the specified value anywhere |
`env=*test` |
Instances where `env` contains `test` (e.g., `test`, `testing`, `pre-test-env`) |
##### Instance Tag Formula Examples
**Simple conditions:**
```
`# Exact match - instances with role exactly equal to "webserver"
role=webserver
# Starts with - instances with app starting with "BANK"
app=^BANK
# Contains - instances with tier containing "PROD"
tier=*PROD`
```
**Complex conditions with AND/OR:**
```
`# Instances in production environment AND with database role
(env=prod AND role=db)
# Instances with app starting with "BANK" OR in the DMZ
(app=^BANK OR zone=dmz)
# Complex: (production databases) OR (any app containing "critical")
(env=prod AND role=db) OR (app=*critical)
# Nested: Instances in prod with apps starting with APP or WEB
(env=prod AND (app=^APP OR app=^WEB))`
```
##### Multi-Value Instance Tags
Instance tags can contain multiple values separated by `|||` (triple-pipe). When matching, the operators check against each individual value in the list.
**Example:** If an instance has the tag `tags=web|||prod|||critical`, then:
- `tags=prod` → matches (exact match on "prod")
- `tags=^pro` → matches ("prod" starts with "pro")
- `tags=*rit` → matches ("critical" contains "rit")
- `tags=dev` → does NOT match (no value equals "dev")
**Note:** You can set multi-value instance tags in INCUS using:
`incus config set <instance> user.tags "web|||prod|||critical"`
The system handles the `user.` prefix internally.
**Migration from NSX:** If you're migrating rules from VMware NSX, the SDN Controller automatically translates NSX security group dynamic membership criteria to these operators:
- NSX `equals` → SDN Controller `=`
- NSX `starts_with` → SDN Controller `=^`
- NSX `contains` → SDN Controller `=*`
Complex NSX security groups with multiple criteria (Match: Any/All) are converted to equivalent AND/OR expressions.
#### Protocol
- **Options**: `TCP`, `UDP`, `ICMP`.
- **Validation**: Must select one (cannot be empty).
- **Example**: `TCP` for HTTP traffic.
#### Port
- **Format**:
- Single port (e.g., `80`).
- Port ranges (e.g., `2000-3000`).
- Comma-separated mix of single ports and ranges (e.g., `80,443,1002-1009,5432`).
- Empty (matches ANY ports, 1-65535, for TCP/UDP).
- **Validation**:
- Optional for `TCP` or `UDP`; if empty, matches all ports (1-65535).
- Disabled and ignored for `ICMP`.
- Must be integers between 1 and 65535, separated by commas (no spaces).
- Invalid examples: `0`, `70000`, `80, 443` (space).
- **Example**: `80,443` for web traffic; leave empty for all ports (1-65535).
**Note:** Omitting the `Port` field for `TCP` or `UDP` protocols will result in the rule applying to ALL ports (1-65535). Explicitly specify ports to restrict the rule's scope.
#### Action
- **Options**: `ALLOW`, `DROP`, `REJECT`.
- **Validation**: Must select one (cannot be empty).
- **Example**: `ALLOW` to permit traffic.
#### Description
- **Format**: Free-text description of the rule's purpose.
- **Validation**: Optional (can be empty).
- **Example**: `Allow HTTP/HTTPS traffic to webservers`.
#### How INCUS Processes Rules
The order in which INCUS applies local and global ACL rules depends on the network's default action:
- **For Networks with Default Action "Drop"**:
INCUS combines all local (`FFSDNIO-`) and global (`FFSDNIOGLB-`) rules applied to a port, applies all `DROP`/`REJECT` rules first, followed by `ALLOW` rules, and finally the default `DROP`.
- **For Networks with Default Action "Allow"**:
INCUS combines all local and global rules, applies all `ALLOW` rules first, followed by `DROP`/`REJECT` rules, and finally the default `ALLOW`.
**Note:** Rule order matters within this framework. Ensure your `ALLOW` and `DROP`/`REJECT` rules are prioritized correctly based on the network's default action and check for global ACLs that might override local ones.
### Step 3: Add More Rules (Optional)
Click "Add another FW rule" to create additional local rules for this cluster. Repeat Step 2 for each new row.
- **Minimum**: At least one valid rule is required.
- **Delete**: Use the "Delete" button on a row to remove it (disabled if only one rule remains).
### Step 4: Review and Submit
Click "Review firewall rules" to validate and preview:
- **Validation Check**:
- Rules: Each has valid `Direction`, `Protocol`, `Action`, non-empty `Source` and `Destination`, and optional `Port` and `Description` per rules above.
- **Review Dialog**: A popup shows each rule's details, including resolved instance tags (e.g., instance names and IPs within the cluster), allowing you to "Commit" or "Cancel".
- **Success**: After committing, redirects to the cluster management page with a confirmation message (e.g., "Firewall rule(s) created successfully").
- **Failure**: Displays an error (e.g., "Invalid source in rule 1: 256.1.2.3").
### Example Firewall Rules
Below are examples of local and global ACLs to illustrate their differences and the use of metadata operators:
**Local ACL with exact match (for cluster `MyCluster`):**
```
`Rule 1:
Direction: Ingress/In
Source: 0.0.0.0/0
Destination: role=webserver
Protocol: TCP
Port: 80,443
Action: ALLOW
Description: Allow HTTP/HTTPS traffic to webservers in MyCluster`
```
**Local ACL with "starts with" operator:**
```
`Rule 1:
Direction: Ingress/In
Source: app=^BANK
Destination: tier=^DB
Protocol: TCP
Port: 5432,3306
Action: ALLOW
Description: Allow BANK* apps to connect to DB* tier instances`
```
**Local ACL with "contains" operator:**
```
`Rule 1:
Direction: Egress/Out
Source: env=*prod
Destination: 10.0.0.0/8
Protocol: TCP
Port: 443
Action: ALLOW
Description: Allow any production-related instances to reach internal network via HTTPS`
```
**Complex ACL with multiple operators and AND/OR logic:**
```
`Rule 1:
Direction: Ingress/In
Source: (env=prod AND zone=dmz) OR (app=^PUBLIC)
Destination: (role=api AND tier=^backend)
Protocol: TCP
Port: 8080,8443
Action: ALLOW
Description: Allow DMZ prod instances and PUBLIC* apps to reach backend API servers`
```
**Global ACL (applies to all clusters):**
```
`Rule 1:
Direction: Egress/Out
Source: env=*prod
Destination: 8.8.8.8,8.8.4.4
Protocol: UDP
Port: 53
Action: ALLOW
Description: Allow DNS queries to Google DNS from any production-related instances across all clusters`
```
### Troubleshooting
- **Error: "Please fill out this field"**: Ensure `Source` and `Destination` are not empty (use `0.0.0.0/0` for ANY).
- **Error: "Please select a value"**: Ensure `Direction`, `Protocol`, and `Action` are selected.
- **Error: "Invalid port value"**: Check that ports are between 1-65535 and comma-separated (e.g., `80,443`, not `80 443`).
- **Error: "Invalid source/destination"**: Verify each comma-separated part is a valid IPv4, CIDR, or instance tag syntax (e.g., `env=prod`).
- **Error: "Invalid condition format"**: Ensure instance tag conditions use valid operators (`=`, `=^`, or `=*`) and have non-empty values after the operator.
- **Rules Not Applying as Expected**:
- Check the network's default action (viewable in "Manage Cluster" under OVN networks) and adjust rule order.
- Verify global ACLs in the "Global ACLs" page, as they may override or conflict with local ACLs.
- If using instance tags, ensure they resolve to IPs (visible in the review popup); empty IP resolutions result in the rule not being applied.
- For `=^` (starts with) and `=*` (contains) operators, verify that your instance tag values actually match the pattern you specified.
### Quick Reference: Instance Tag Operator Syntax
|
Use Case |
Syntax |
Example |
|
Match exact value |
`key=value` |
`env=prod` |
|
Match values starting with prefix |
`key=^prefix` |
`app=^BANK` |
|
Match values containing substring |
`key=*substring` |
`tier=*PROD` |
|
AND - both conditions must match |
`(cond1 AND cond2)` |
`(env=prod AND role=db)` |
|
OR - either condition can match |
`(cond1 OR cond2)` |
`(env=prod OR env=staging)` |
|
Complex nested conditions |
`((cond1 AND cond2) OR cond3)` |
`((env=prod AND role=web) OR app=^PUBLIC)` |
**Note:** Instance tag formulas (e.g., `role=webserver`) match instance tags set in INCUS (e.g., via `incus config set <instance> user.role webserver`). The system handles the `user.` prefix internally. For global ACLs, instance tags apply across all clusters, so ensure consistency in tag naming.
---
## OVNHAproxyLB
*Source: `/static/OVNHAproxyLB.html`*
HAProxy Load Balancer Management Guide - AETHER
## HAProxy Load Balancer Management Guide
This guide explains how to use the HAProxy Load Balancer Management page in AETHER. HAProxy provides high-availability load balancing for your applications running on INCUS clusters with OVN networking.
#### Table of Contents
- 1. Overview and Architecture
- 2. Prerequisites
- 3. Accessing the HAProxy Management Page
- 4. Managing HAProxy Base Images
- 4.1 Building a New Image
- 4.2 Pushing Images to Clusters
- 4.3 Managing Image Versions
- 5. Deploying HAProxy Infrastructure
- 5.1 Deployment Process
- 5.2 Deployment Fields Explained
- 5.3 Deleting Infrastructure
- 6. Managing VIP Addresses
- 6.1 Adding a VIP
- 6.2 Deleting a VIP
- 7. Configuring Services
- 7.1 Creating a Service
- 7.2 Service Fields Explained
- 7.3 Understanding Service Modes
- 7.4 Load Balancing Methods
- 7.5 Health Checks
- 7.6 Sticky Sessions
- 7.7 Configuring Backend Servers
- 7.8 Editing a Service
- 7.9 Deleting a Service
- 7.10 Monitoring Service Health and Statistics
- 7.11 Advanced Options
- 7.11.1 Security Options
- 7.11.2 Performance Options
- 7.11.3 Rate Limiting
- 7.11.4 Access Control
- 8. Example Configurations
- 9. Troubleshooting
- 10. Glossary
### 1. Overview and Architecture
The AETHER HAProxy Load Balancer provides enterprise-grade load balancing for applications running on INCUS clusters. It automatically deploys and manages a highly available HAProxy setup with the following architecture:
Internet / Client Traffic
|
v
+-------------------------+
| OVN Load Balancer |
| (Virtual IP - VIP) |
+-------------------------+
/ \
/ \
v v
+----------------+ +----------------+
| HAProxy 01 | | HAProxy 02 |
| (Instance 1) | | (Instance 2) |
+----------------+ +----------------+
\ /
\ /
v v
+-------+ +-------+ +-------+
|Backend| |Backend| |Backend|
| 1 | | 2 | | 3 |
+-------+ +-------+ +-------+
#### Key Components:
- **OVN Load Balancer (VIP)**: The entry point for all traffic. The Virtual IP (VIP) is the address clients connect to. OVN distributes traffic between the two HAProxy instances for high availability.
- **HAProxy Instances**: Two identical HAProxy containers (01 and 02) run on your cluster. If one fails, the other continues serving traffic seamlessly.
- **Backend Servers**: Your actual application servers that HAProxy forwards traffic to based on your configured rules.
**Note:** Each INCUS cluster can have one HAProxy infrastructure deployment. Multiple services can share this infrastructure using different VIPs and ports.
### 2. Prerequisites
Before you can use the HAProxy Load Balancer functionality, ensure the following:
#### Permissions Required
- **AETHER Full Admin** or **AETHER LB Admin** role is required to:
- Build and manage HAProxy base images
- Deploy or delete HAProxy infrastructure
- Add or delete VIPs
- Create, edit, or delete services
#### Technical Requirements
- An INCUS cluster must be registered in AETHER
- The cluster must have at least one OVN network configured
- Internet access from the cluster is required to build HAProxy images (for package downloads)
- Available IP addresses on the OVN network for the HAProxy instances and VIP(s)
### 3. Accessing the HAProxy Management Page
To access the HAProxy Load Balancer management page:
- Log in to the AETHER web interface
- Navigate to **"HAProxy"** in the main navigation menu
- The page displays:
- **HAProxy Base Images** section (admin only) - for managing HAProxy container images
- **Select Cluster to manage LB** dropdown - to choose which cluster to configure
- **Infrastructure, VIPs, and Services** sections - appear after selecting a cluster with deployed infrastructure
### 4. Managing HAProxy Base Images
Before deploying HAProxy infrastructure, you need a HAProxy base image available on your target cluster. The base image is a pre-configured container image with HAProxy installed.
#### 4.1 Building a New Image
To build a new HAProxy base image:
- Click the **"Build New Image"** button in the HAProxy Base Images section
- Select the **Build on Cluster** - choose any cluster with internet access
- Enter a **Version** number (e.g., "1.0.0", "2.1.0")
- Click **"Build Image"**
**Warning:** The build process takes 5-10 minutes as it downloads and installs packages. Do not close the browser during this process. The build progress is displayed in real-time.
The build process:
- Creates a temporary container on the selected cluster
- Installs HAProxy and required packages
- Configures the base HAProxy settings
- Creates an image from the container
- Stores the image in the AETHER database
- Cleans up the temporary container
#### 4.2 Pushing Images to Clusters
After building an image, you must push it to each cluster where you want to deploy HAProxy infrastructure:
- In the images table, find the version you want to push
- Click **"Push to Cluster"**
- Select the **Target Cluster** from the dropdown
- Click **"Push Image"**
**Note:** An image must be pushed to a cluster before you can deploy HAProxy infrastructure on that cluster. The cluster selection dropdown will indicate "(No Image)" for clusters without the HAProxy image.
#### 4.3 Managing Image Versions
The images table shows all available HAProxy images with the following actions:
|
Action |
Description |
|
**Set Current** |
Marks this version as the default for new deployments. Only one version can be "Current" at a time. |
|
**Push to Cluster** |
Copies the image to a specific cluster's image store. |
|
**Delete** |
Removes the image from the AETHER database. Does not affect already-deployed HAProxy instances. |
### 5. Deploying HAProxy Infrastructure
HAProxy infrastructure consists of two HAProxy container instances and an OVN load balancer that provides failover between them.
#### 5.1 Deployment Process
- Select a cluster from the **"Select Cluster to manage LB"** dropdown
- If no infrastructure exists, you'll see a message with a **"Deploy HAProxy Infrastructure"** button
- Click the button to open the deployment form
- Fill in all required fields (explained below)
- Click **"Deploy"**
**What happens during deployment:**
- Two HAProxy container instances are created (ffsdn-haproxy-[clusterID]-01 and -02)
- Both instances are started and configured with a base HAProxy configuration
- An OVN load balancer is created to distribute traffic to both instances
- Firewall rules are automatically configured to allow traffic
- A default VIP is created for your services
#### 5.2 Deployment Fields Explained
##### OVN Network
Select the OVN network where the HAProxy instances will be deployed. This should be a network that:
- Has connectivity to your backend servers
- Has available IP addresses for the HAProxy instances and VIP(s)
- Is accessible from where your clients will connect
**Example:** `ovn-production`, `ovn-dmz`
##### Load Balancer VIP
The Virtual IP address that clients will connect to. This IP is managed by the OVN load balancer and routes traffic to the HAProxy instances.
- Select an available IP from the **dropdown menu**
- The dropdown shows unused IP addresses from the configured VIP range on the selected OVN network
- This becomes the "front door" for your load-balanced services
**Note:** The available IPs are automatically detected from the OVN network's VIP range configuration. No manual IP entry is required.
##### HAProxy 01 IP and HAProxy 02 IP
The IP addresses assigned to each HAProxy container instance. These are internal addresses used by the OVN load balancer.
- Select available IPs from the **dropdown menus**
- The dropdowns show unused IP addresses from the selected OVN network
- Each HAProxy instance must have a unique IP address
- Both IPs should be different from the VIP address
**Note:** Available IPs are automatically detected from the OVN network. Simply select different IPs for each HAProxy instance from the dropdown menus.
##### CPU Limit
The number of CPU cores allocated to each HAProxy instance.
- Default: `2`
- For high-traffic environments, consider increasing this value
- HAProxy is very CPU-efficient; 2 cores handle most workloads
##### Memory Limit
The amount of RAM allocated to each HAProxy instance.
- Default: `1GB`
- For environments with many concurrent connections, consider `2GB` or more
- Format: Use suffix like `512MB`, `1GB`, `2GB`
#### 5.3 Deleting Infrastructure
To completely remove HAProxy infrastructure from a cluster:
- Select the cluster from the dropdown
- Click the **"Delete Infrastructure"** button (red)
- Confirm the deletion in the popup dialog
**Warning:** Deleting infrastructure will:
- Stop and delete both HAProxy container instances
- Remove all VIPs and their OVN load balancers
- Delete all configured services
- Remove associated firewall rules
This action cannot be undone. All traffic to your services will be interrupted immediately.
### 6. Managing VIP Addresses
A VIP (Virtual IP) is the address that clients connect to. You can have multiple VIPs on a single HAProxy infrastructure, allowing you to serve different applications or environments.
#### 6.1 Adding a VIP
After deploying infrastructure, you can add additional VIPs:
- In the **VIP Addresses** section, click **"+ Add VIP"**
- Enter a **VIP Name** - a descriptive name (e.g., "Production HTTPS", "API Gateway")
- Enter the **IP Address** - must be unused on the OVN network
- Click **"Add VIP"**
**When to use multiple VIPs:**
- Separating production and staging traffic
- Different external IP addresses for different applications
- Isolating traffic for security or compliance reasons
- Different SSL certificates for different domains
#### 6.2 Deleting a VIP
- Find the VIP in the **VIP Addresses** table
- Click the **"Delete"** button
- Confirm the deletion
**Warning:** Deleting a VIP will also delete all services associated with that VIP. Ensure you have migrated or backed up any important configurations first.
### 7. Configuring Services
Services define how traffic is handled by HAProxy. Each service listens on a specific VIP and port combination and forwards traffic to one or more backend servers.
#### 7.1 Creating a Service
- In the **Services** section, click **"+ Add Service"**
- Fill in the service configuration (explained in detail below)
- Add at least one backend server
- Click **"Create Service"**
#### 7.2 Service Fields Explained
##### Service Name
A unique, descriptive name for the service. This appears in the HAProxy configuration and logs.
- Use lowercase letters, numbers, and hyphens
- Should be descriptive of what the service does
- Must be unique within the infrastructure
**Examples:** `customer-web`, `api-gateway`, `internal-db-proxy`
##### Description
Optional free-text description of the service's purpose. Helpful for documentation and team communication.
**Example:** "Main customer-facing website load balancer"
##### VIP
Select which VIP this service should listen on. The dropdown shows all VIPs configured for this infrastructure.
Multiple services can share the same VIP if they use different ports or hostnames.
##### Listen Port
The TCP port that HAProxy will listen on for this service.
- Common ports: 80 (HTTP), 443 (HTTPS), 8080, 8443
- Must be unique per VIP (unless using hostname-based routing)
- Range: 1-65535 (ports below 1024 are typically reserved)
**Examples:** `443`, `8080`, `3306`
##### Hostname (SNI/Host header)
Optional hostname for routing traffic based on the requested domain name. This enables multiple services to share the same VIP and port.
- For HTTPS Passthrough mode: Routes based on SNI (Server Name Indication)
- For HTTP mode: Routes based on the Host header
- Leave empty if not using hostname-based routing
**Examples:** `www.example.com`, `api.example.com`, `*.example.com`
#### 7.3 Understanding Service Modes
The **Mode** determines how HAProxy processes traffic. Choose the appropriate mode based on your application's requirements:
|
Mode |
Layer |
Use Case |
SSL/TLS Handling |
|
**TCP (Layer 4)** |
Transport |
Database connections, generic TCP services, any non-HTTP protocol |
Passed through unchanged |
|
**HTTP** |
Application |
Plain HTTP websites, internal APIs without encryption |
Not applicable (no SSL) |
|
**HTTPS Passthrough (SNI routing)** |
Transport |
HTTPS where backend servers handle SSL termination |
Passed through to backend (HAProxy reads SNI only) |
|
**HTTPS Termination** |
Application |
HTTPS where HAProxy handles SSL certificates |
Terminated at HAProxy (requires certificate upload) |
##### TCP (Layer 4) - Detailed Explanation
In TCP mode, HAProxy operates at the transport layer and forwards raw TCP connections without inspecting the content.
**Best for:**
- Database load balancing (MySQL, PostgreSQL, MongoDB)
- Message queues (RabbitMQ, Kafka)
- Custom TCP protocols
- Any application where HAProxy shouldn't modify the traffic
**Limitations:** Cannot route based on HTTP headers or URLs.
##### HTTP - Detailed Explanation
In HTTP mode, HAProxy understands HTTP protocol and can make routing decisions based on headers, URLs, and cookies.
**Best for:**
- Internal HTTP APIs
- Development/testing environments
- Services behind a separate SSL termination point
**Features available:** URL-based routing, header manipulation, cookie-based persistence.
##### HTTPS Passthrough (SNI Routing) - Detailed Explanation
HAProxy reads the SNI (Server Name Indication) from the TLS handshake to route traffic, but does not decrypt it. The SSL/TLS connection is passed through to the backend server.
**Best for:**
- When backend servers must handle their own SSL certificates
- End-to-end encryption requirements
- Multiple HTTPS sites on the same IP/port with different certificates
**How it works:** When a client connects, HAProxy reads the hostname from the SNI extension (sent in cleartext during TLS handshake), routes to the appropriate backend, and the backend completes the TLS handshake.
##### HTTPS Termination - Detailed Explanation
HAProxy terminates the SSL/TLS connection, decrypts the traffic, and can then forward it as HTTP or re-encrypt it to backends.
**Best for:**
- Centralized certificate management
- When you need to inspect or modify HTTPS traffic
- Offloading SSL processing from backend servers
**Requires:** SSL certificate and private key to be uploaded when creating the service.
#### 7.4 Load Balancing Methods
The **Balance Method** determines how HAProxy distributes requests among backend servers:
|
Method |
Algorithm |
Best For |
|
**Round Robin** |
Distributes requests sequentially to each backend in turn |
General purpose, when all backends have similar capacity |
|
**Least Connections** |
Sends new requests to the backend with fewest active connections |
Long-lived connections, varying request complexity |
|
**Source IP Hash** |
Routes requests from the same client IP to the same backend |
Simple session affinity without cookies |
|
**First Available** |
Uses the first backend with available connection slots |
Minimizing the number of active backends |
##### Round Robin - When to Use
The default and most common method. Each backend receives requests in sequence.
- Ideal when all backend servers have equal capacity
- Works well for stateless applications
- Simple and predictable distribution
**Example:** 3 backends receive requests in order: 1, 2, 3, 1, 2, 3, ...
##### Least Connections - When to Use
Dynamically routes to the backend handling the fewest requests.
- Best for requests with varying processing times
- Prevents overloading slower backends
- Ideal for WebSocket or long-polling connections
**Example:** If Backend 1 has 10 connections and Backend 2 has 5, new requests go to Backend 2.
##### Source IP Hash - When to Use
Creates a hash of the client's IP address to consistently route them to the same backend.
- Provides session persistence without cookies
- Useful when you can't use application-level session management
- Note: Adding/removing backends will redistribute some clients
**Example:** Client 192.168.1.100 always goes to Backend 2 (based on hash).
##### First Available - When to Use
Fills the first backend to capacity before using the next one.
- Useful for minimizing resource usage
- Allows backends to be powered down when not needed
- Not recommended for even load distribution
#### 7.5 Health Checks
When **Health Check** is enabled, HAProxy periodically verifies that backend servers are healthy and removes unhealthy ones from rotation.
**Recommendation:** Always enable health checks in production environments. This ensures traffic is only sent to working backends.
Health check behavior:
- HAProxy sends periodic probes to each backend (default: every 5 seconds)
- For TCP mode: Verifies the port accepts connections
- For HTTP/HTTPS modes: Can verify specific paths return expected responses
- Unhealthy backends are temporarily removed from rotation
- Backends are automatically re-added when they recover
#### 7.6 Sticky Sessions (Session Persistence)
When **Sticky Sessions** is enabled, HAProxy ensures that requests from the same client are sent to the same backend server.
|
Feature |
Description |
|
**How it works** |
HAProxy inserts a cookie into the response that identifies the backend server |
|
**Cookie name** |
SERVERID (automatically managed) |
|
**Duration** |
Session-based (expires when browser closes) or configurable |
**When to use sticky sessions:**
- Applications that store session data locally (not in shared storage)
- Shopping carts, login sessions, multi-step forms
- When backend affinity improves cache hit rates
**Warning:** Sticky sessions can lead to uneven load distribution if some clients generate more traffic than others. Consider using shared session storage (Redis, database) instead when possible.
#### 7.7 Configuring Backend Servers
Backend servers are the actual application servers that receive traffic from HAProxy. Each service must have at least one backend.
##### Backend Fields:
|
Field |
Description |
Example |
|
**Backend Name** |
Unique identifier for this backend within the service. Used in logs and statistics. |
`web-server-1`, `api-node-a` |
|
**IP Address** |
The IP address of the backend server. Must be reachable from the HAProxy instances. |
`10.50.1.100` |
|
**Port** |
The port on the backend server where the application listens. |
`8080`, `3000`, `443` |
|
**Weight** |
Relative weight for load distribution. Higher weight = more traffic. Default: 100. |
`100` (normal), `200` (double traffic), `50` (half traffic) |
##### Adding Multiple Backends
- Click **"+ Add Backend Server"** to add another backend row
- Fill in the backend details
- Repeat for each backend server
- Use the **"x"** button to remove a backend row
##### Weight Examples
Weights are relative, not absolute percentages:
- **Equal distribution:** All backends with weight 100
- **2:1 ratio:** Backend A: 200, Backend B: 100 (A gets twice the traffic)
- **Gradual rollout:** Old server: 90, New server: 10 (10% to new server)
- **Drain server:** Set weight to 0 (no new connections, existing ones continue)
#### 7.8 Editing a Service
- Find the service card in the **Services** section
- Click the **"Edit"** button
- Modify the desired fields
- Add, remove, or modify backend servers as needed
- Click **"Save Changes"**
**Note:** The VIP cannot be changed when editing a service. To move a service to a different VIP, delete it and create a new one.
#### 7.9 Deleting a Service
- Find the service card in the **Services** section
- Click the **"Delete"** button (red)
- Confirm the deletion in the popup dialog
**Warning:** Deleting a service immediately stops traffic routing. Ensure you have communicated the change to affected users/systems.
#### 7.10 Monitoring Service Health and Statistics
Each service has a **"Health & Stats"** button that opens a detailed monitoring popup. This provides real-time visibility into service health, traffic statistics, and HAProxy node resource usage.
##### Accessing the Health & Stats Modal
- Find the service card in the **Services** section
- Click the **"Health & Stats"** button
- The modal displays four sections of information (described below)
- Click **"Refresh"** to update the data, or **"Close"** to exit
##### Section 1: HAProxy Backend Server Statistics
This section shows the health status of each backend server **as determined by HAProxy's health checks**. This is the authoritative source for whether HAProxy considers a backend healthy and is routing traffic to it.
|
Column |
Description |
|
**Server** |
The backend server name as configured |
|
**Address** |
The IP address and port of the backend server (e.g., 10.50.1.10:8080) |
|
**Status** |
HAProxy's health verdict: **UP** (healthy, receiving traffic) or **DOWN** (unhealthy, removed from rotation) |
|
**Sessions** |
Current active sessions / maximum sessions seen |
|
**Bytes In/Out** |
Total traffic received from and sent to this backend |
|
**Weight** |
The configured weight for load distribution |
|
**Last Check** |
Result of the most recent health check and response time in milliseconds |
**Note:** A backend showing **DOWN** means HAProxy's configured health check failed. The backend will not receive new traffic until it passes health checks again.
##### Section 2: HAProxy Statistics
This section shows traffic statistics from HAProxy's perspective, split into Frontend and Backend summaries.
###### Frontend Statistics
Shows statistics for the HAProxy frontend listener on this service's port.
**Important:** If multiple services share the same port (using hostname-based routing), the Frontend statistics show **combined traffic for all services on that port**, not just this specific service.
|
Metric |
Description |
|
**Current Sessions** |
Number of active client connections right now |
|
**Max Sessions** |
Highest number of concurrent sessions seen |
|
**Total Sessions** |
Cumulative number of sessions since HAProxy started |
|
**Session Rate** |
New sessions per second |
|
**Bytes In/Out** |
Total traffic received from clients and sent to clients |
###### Backend Summary
Shows aggregated statistics for this specific service's backend group only.
|
Metric |
Description |
|
**Status** |
Overall backend status (UP if at least one server is healthy) |
|
**Current Sessions** |
Active connections to backend servers |
|
**Total Sessions** |
Cumulative sessions handled by this backend |
|
**Bytes In/Out** |
Traffic to and from backend servers |
|
**Downtime** |
Total time the backend has been unavailable |
##### Section 3: HAProxy Node Resources
This section displays CPU and memory usage for both HAProxy container instances (HAProxy 01 and HAProxy 02). This helps you monitor whether the HAProxy nodes have sufficient resources.
|
Metric |
Description |
|
**CPUs Assigned** |
Number of CPU cores allocated to the instance |
|
**CPU Usage** |
Current CPU utilization percentage |
|
**Memory Assigned** |
Amount of RAM allocated to the instance |
|
**Memory Usage** |
Current memory utilization (amount and percentage) |
###### Resource Usage Color Thresholds
CPU and memory usage are color-coded to help you quickly identify resource constraints:
|
Color |
Usage Range |
Interpretation |
|
**Green** |
Below 50% |
Healthy - plenty of headroom |
|
**Orange** |
50% - 79% |
Moderate - monitor for increases |
|
**Red** |
80% and above |
High - consider increasing resources |
**Tip:** If you consistently see orange or red resource usage, consider increasing the CPU or memory limits for the HAProxy infrastructure. This can be done by deleting and redeploying the infrastructure with higher resource allocations.
#### 7.11 Advanced Options
The Advanced Options section provides additional features for security hardening, performance optimization, rate limiting, and access control. These options are available when creating or editing a service and are found in a collapsible section below the basic service configuration.
**Note:** Advanced Options only apply to services using **HTTP** or **HTTPS Termination** modes. They have no effect on TCP or HTTPS Passthrough services, as those modes operate at Layer 4 without inspecting HTTP content.
##### 7.11.1 Security Options
Security options help protect your applications from common web attacks and enforce security best practices through HTTP headers.
|
Option |
Description |
When to Use |
|
**Force HTTPS Redirect** |
Automatically redirects HTTP requests to HTTPS using a 301 (permanent) redirect. |
Enable on HTTP services that should only be accessed via HTTPS. Ensures users always use the encrypted version. |
|
**Enable HSTS** |
Adds the `Strict-Transport-Security` header, instructing browsers to only connect via HTTPS for a specified duration. |
Enable for production HTTPS services to prevent protocol downgrade attacks and cookie hijacking. |
|
**Clickjacking Protection** |
Adds the `X-Frame-Options: SAMEORIGIN` header, preventing your site from being embedded in iframes on other domains. |
Enable for web applications to prevent clickjacking attacks where attackers overlay invisible frames. |
|
**Prevent MIME Sniffing** |
Adds the `X-Content-Type-Options: nosniff` header, preventing browsers from guessing content types. |
Enable to prevent MIME confusion attacks where malicious files are executed as scripts. |
###### Force HTTPS Redirect - Detailed
When enabled, any request arriving over plain HTTP will receive a `301 Moved Permanently` response redirecting to the HTTPS version of the URL.
**Example:**
```
`Request: GET http://example.com/page
Response: 301 Redirect to https://example.com/page`
```
**Warning:** Only enable this on services where HTTPS is properly configured. Enabling on a plain HTTP-only service will create a redirect loop.
###### HSTS (HTTP Strict Transport Security) - Detailed
HSTS tells browsers to remember that your site should only be accessed via HTTPS. Once a browser receives the HSTS header, it will automatically convert any HTTP links to HTTPS before making the request.
**Duration Options:**
|
Setting |
Duration |
Recommended Use |
|
1 year (recommended) |
31,536,000 seconds |
Production services with stable HTTPS configuration |
|
6 months |
15,768,000 seconds |
Production services, slightly more conservative |
|
30 days |
2,592,000 seconds |
Staging environments or services with changing configurations |
|
1 day (testing) |
86,400 seconds |
Testing HSTS before committing to longer durations |
**Important:** Once HSTS is enabled with a long duration, browsers will refuse to connect via HTTP until the timer expires. Start with shorter durations to test, then increase for production.
###### Clickjacking Protection - Detailed
Clickjacking is an attack where a malicious site embeds your site in an invisible iframe and tricks users into clicking hidden buttons. The `X-Frame-Options: SAMEORIGIN` header prevents this by only allowing your site to be framed by pages on the same domain.
**Effect:**
- Your site CAN be embedded in iframes on your own domain
- Your site CANNOT be embedded in iframes on other domains
###### MIME Sniffing Protection - Detailed
Some browsers try to "guess" the content type of a response, which can be exploited to execute malicious content. The `X-Content-Type-Options: nosniff` header prevents this behavior.
**Example attack prevented:** An attacker uploads a file named `image.jpg` that actually contains JavaScript. Without this protection, the browser might execute it as a script.
##### 7.11.2 Performance Options
Performance options help reduce bandwidth usage, improve response times, and provide better debugging information to backend servers.
|
Option |
Description |
When to Use |
|
**Enable Compression (gzip)** |
Compresses text-based responses (HTML, CSS, JavaScript, JSON, XML) before sending to clients. |
Enable for web applications serving text content. Can significantly reduce bandwidth usage. |
|
**Forward Client IP** |
Adds `X-Real-IP`, `X-Forwarded-For`, and `X-Forwarded-Proto` headers containing the original client's information. |
Enable when your backend applications need to know the real client IP address for logging, geo-location, or security purposes. |
|
**Hide Server Version Info** |
Removes the `Server` and `X-Powered-By` headers from responses. |
Enable to hide information about your server software from potential attackers. |
###### Compression - Detailed
When enabled, HAProxy will compress responses using gzip for the following content types:
- `text/html` - HTML pages
- `text/plain` - Plain text files
- `text/css` - Stylesheets
- `text/javascript` and `application/javascript` - JavaScript files
- `application/json` - JSON API responses
- `application/xml` - XML documents
**Typical bandwidth savings:** 60-80% for text-based content.
**Note:** Images, videos, and already-compressed files (ZIP, PDF) are not compressed as they wouldn't benefit from additional compression.
###### Forward Client IP - Detailed
When traffic passes through a load balancer, backend servers see HAProxy's IP address instead of the client's real IP. This option adds headers to preserve the original client information.
**Headers Added:**
|
Header |
Value |
Example |
|
`X-Real-IP` |
Client's IP address |
`203.0.113.50` |
|
`X-Forwarded-For` |
Client's IP address (chain of proxies) |
`203.0.113.50` |
|
`X-Forwarded-Proto` |
Original protocol (http or https) |
`https` |
**Backend Configuration:** Your application needs to be configured to read these headers. Most web frameworks have settings for "trust proxy" or "forwarded headers".
###### Hide Server Info - Detailed
By default, web servers include headers identifying the software and version (e.g., `Server: nginx/1.18.0`). Attackers can use this information to target known vulnerabilities.
**Headers Removed:**
- `Server` - Identifies web server software
- `X-Powered-By` - Identifies application framework (e.g., PHP, ASP.NET)
This is a "security through obscurity" measure - it makes reconnaissance slightly harder but shouldn't be your only security measure.
##### 7.11.3 Rate Limiting
Rate limiting protects your services from abuse, denial-of-service attacks, and runaway clients by limiting how many requests a single IP address can make.
|
Option |
Description |
|
**Enable Rate Limiting** |
Activates request rate tracking and limiting per client IP address. |
|
**Max Requests per 10 Seconds** |
The maximum number of requests allowed from a single IP address within a 10-second window. Default: 100. Range: 1-10,000. |
###### How Rate Limiting Works
HAProxy tracks the request rate for each client IP address using a sliding window of 10 seconds. When a client exceeds the configured limit, they receive a `429 Too Many Requests` response.
**Example with 100 requests per 10 seconds:**
- Client makes 100 requests in 5 seconds - All succeed
- Client makes request #101 at second 6 - Receives 429 error
- After second 10, the counter resets based on the sliding window
###### Choosing the Right Limit
|
Limit |
Equivalent Rate |
Suitable For |
|
10 |
~1 req/sec |
API endpoints with strict limits, login pages |
|
100 (default) |
~10 req/sec |
Standard web applications, moderate API usage |
|
500 |
~50 req/sec |
High-traffic web pages with many assets |
|
1,000 |
~100 req/sec |
CDN-like usage, API-heavy applications |
|
10,000 |
~1,000 req/sec |
Very high traffic, internal services |
**Warning:** Setting limits too low may block legitimate users, especially if multiple users share a single IP address (corporate networks, NAT). Start with higher limits and reduce based on monitoring.
##### 7.11.4 Access Control
Access control options allow you to block requests based on URL paths or User-Agent headers, providing a simple layer of protection against unwanted traffic.
|
Option |
Description |
Format |
|
**Block Paths** |
Denies access to requests matching specified URL path prefixes. |
Comma-separated list of paths (e.g., `/admin, /wp-admin, /.git`) |
|
**Block User Agents** |
Denies access to requests with User-Agent headers containing specified strings. |
Comma-separated list of strings (e.g., `crawler, badbot, scanner`) |
###### Blocked Paths - Detailed
Requests to blocked paths receive a `403 Forbidden` response. The matching is based on path prefix, so blocking `/admin` will also block `/admin/users`, `/admin/settings`, etc.
**Common paths to block:**
|
Path |
Reason |
|
`/admin` |
Administrative interfaces (if not needed publicly) |
|
`/wp-admin`, `/wp-login.php` |
WordPress admin areas (frequent attack target) |
|
`/.git` |
Git repository data (should never be public) |
|
`/.env` |
Environment configuration files |
|
`/phpmyadmin` |
Database management interface |
|
`/config` |
Configuration directories |
**Example configuration:** `/admin, /wp-admin, /.git, /.env, /backup`
###### Blocked User Agents - Detailed
Requests from clients with User-Agent strings containing any of the specified strings will receive a `403 Forbidden` response. The matching is case-insensitive and checks if the string appears anywhere in the User-Agent.
**Common user agents to block:**
|
String |
Reason |
|
`sqlmap` |
SQL injection scanning tool |
|
`nikto` |
Web vulnerability scanner |
|
`nmap` |
Network scanner |
|
`masscan` |
Port scanner |
|
`ZmEu` |
Known malicious scanner |
|
`wget`, `curl` |
Command-line tools (block if not needed for legitimate API access) |
**Example configuration:** `sqlmap, nikto, nmap, ZmEu, masscan`
**Warning:** User-Agent strings can easily be spoofed. This is a deterrent against casual/automated attacks, not a security guarantee. Sophisticated attackers will use legitimate-looking User-Agents.
###### Access Control Best Practices
- **Defense in depth:** Don't rely solely on HAProxy access control. Also configure your application and firewall.
- **Test before production:** Verify blocked paths don't affect legitimate functionality.
- **Monitor logs:** Review blocked requests to identify attack patterns and adjust rules.
- **Keep it simple:** Start with a small set of critical paths and expand as needed.
### 8. Example Configurations
#### Example 1: Simple Web Application (HTTP)
```
`Service Name: company-website
Description: Main company website
VIP: 192.168.1.200
Listen Port: 80
Mode: HTTP
Balance Method: Round Robin
Health Check: Enabled
Sticky Sessions: Disabled
Backends:
- web-server-1: 10.50.1.10:8080 (weight: 100)
- web-server-2: 10.50.1.11:8080 (weight: 100)
- web-server-3: 10.50.1.12:8080 (weight: 100)`
```
#### Example 2: HTTPS Website with SSL Passthrough
```
`Service Name: secure-portal
Description: Customer portal with end-to-end encryption
VIP: 192.168.1.200
Listen Port: 443
Hostname: portal.example.com
Mode: HTTPS Passthrough (SNI routing)
Balance Method: Least Connections
Health Check: Enabled
Sticky Sessions: Enabled
Backends:
- portal-node-1: 10.50.2.20:443 (weight: 100)
- portal-node-2: 10.50.2.21:443 (weight: 100)`
```
#### Example 3: Database Load Balancing (TCP)
```
`Service Name: mysql-cluster
Description: MySQL read replicas for reporting
VIP: 192.168.1.201
Listen Port: 3306
Mode: TCP (Layer 4)
Balance Method: Least Connections
Health Check: Enabled
Sticky Sessions: Disabled
Backends:
- mysql-replica-1: 10.50.3.30:3306 (weight: 100)
- mysql-replica-2: 10.50.3.31:3306 (weight: 100)
- mysql-replica-3: 10.50.3.32:3306 (weight: 50) # Less powerful server`
```
#### Example 4: API Gateway with Gradual Rollout
```
`Service Name: api-v2-rollout
Description: API with gradual traffic shift to v2
VIP: 192.168.1.200
Listen Port: 8443
Hostname: api.example.com
Mode: HTTPS Passthrough (SNI routing)
Balance Method: Round Robin
Health Check: Enabled
Sticky Sessions: Disabled
Backends:
- api-v1-node-1: 10.50.4.40:8443 (weight: 45)
- api-v1-node-2: 10.50.4.41:8443 (weight: 45)
- api-v2-node-1: 10.50.4.50:8443 (weight: 10) # 10% of traffic to new version`
```
#### Example 5: Multiple Services on Same VIP
```
`VIP: 192.168.1.200
Service 1: website-http
Listen Port: 80
Mode: HTTP
Backends: web-1:8080, web-2:8080
Service 2: website-https
Listen Port: 443
Mode: HTTPS Passthrough
Backends: web-1:8443, web-2:8443
Service 3: api-service
Listen Port: 8080
Mode: HTTP
Backends: api-1:3000, api-2:3000, api-3:3000`
```
#### Example 6: Security-Hardened Web Application with Advanced Options
```
`Service Name: secure-webapp
Description: Production web application with full security hardening
VIP: 192.168.1.200
Listen Port: 443
Hostname: app.example.com
Mode: HTTPS Termination
Balance Method: Least Connections
Health Check: Enabled
Sticky Sessions: Enabled
Advanced Options - Security:
Force HTTPS Redirect: Enabled
HSTS: Enabled (1 year)
Clickjacking Protection: Enabled
MIME Sniffing Protection: Enabled
Advanced Options - Performance:
Compression: Enabled
Forward Client IP: Enabled
Hide Server Info: Enabled
Advanced Options - Rate Limiting:
Rate Limiting: Enabled
Max Requests per 10 seconds: 200
Advanced Options - Access Control:
Block Paths: /admin, /.git, /.env, /backup, /config
Block User Agents: sqlmap, nikto, ZmEu, masscan
Backends:
- app-node-1: 10.50.5.10:8080 (weight: 100)
- app-node-2: 10.50.5.11:8080 (weight: 100)
- app-node-3: 10.50.5.12:8080 (weight: 100)`
```
#### Example 7: API Gateway with Rate Limiting
```
`Service Name: public-api
Description: Public API with rate limiting protection
VIP: 192.168.1.202
Listen Port: 443
Hostname: api.example.com
Mode: HTTPS Termination
Balance Method: Round Robin
Health Check: Enabled
Sticky Sessions: Disabled
Advanced Options - Security:
HSTS: Enabled (1 year)
MIME Sniffing Protection: Enabled
Advanced Options - Performance:
Compression: Enabled (for JSON responses)
Forward Client IP: Enabled
Hide Server Info: Enabled
Advanced Options - Rate Limiting:
Rate Limiting: Enabled
Max Requests per 10 seconds: 100 (10 req/sec average)
Advanced Options - Access Control:
Block Paths: /internal, /debug, /health/detailed
Block User Agents: curl, wget (if blocking CLI tools)
Backends:
- api-server-1: 10.50.6.20:3000 (weight: 100)
- api-server-2: 10.50.6.21:3000 (weight: 100)`
```
### 9. Troubleshooting
#### Common Issues and Solutions
##### Issue: "HAProxy base image not available on cluster"
**Cause:** The HAProxy image hasn't been pushed to the selected cluster.
**Solution:**
- Go to the "HAProxy Base Images" section
- Click "Push to Cluster" on the desired image version
- Select the target cluster and push the image
- Wait for the push to complete, then retry deployment
##### Issue: "Failed to create HAProxy instance"
**Possible causes:**
- IP address already in use
- Insufficient resources on the cluster
- Network connectivity issues
**Solution:**
- Verify the IP addresses are not already assigned to other instances
- Check cluster resource availability (CPU, memory)
- Verify the OVN network is properly configured
##### Issue: Service is created but traffic is not reaching backends
**Troubleshooting steps:**
- Verify the HAProxy instances are running (check status badges)
- Confirm backend servers are running and accessible on the specified ports
- Check that firewall rules allow traffic from HAProxy IPs to backend IPs
- Verify the VIP is correctly configured in DNS or client applications
- Use "Reload Config" button to ensure configuration is applied
##### Issue: Health checks failing
**Possible causes:**
- Backend server is down or not responding
- Firewall blocking health check traffic from HAProxy instances
- Incorrect backend port configuration
- Application taking too long to respond
**Solution:**
- Test connectivity from the cluster network to the backend
- Verify the backend application is listening on the configured port
- Check backend server logs for errors
- Ensure ACLs allow traffic from HAProxy IPs to backend servers
##### Issue: Sticky sessions not working
**Possible causes:**
- Client not accepting cookies
- Using TCP mode (cookies only work in HTTP/HTTPS modes)
- Multiple levels of proxies stripping cookies
**Solution:**
- Verify the service is using HTTP or HTTPS mode (not TCP)
- Check that client browsers accept cookies
- If using HTTPS Passthrough, sticky sessions use different mechanisms
##### Issue: Build image process fails
**Possible causes:**
- No internet access from the cluster
- Package repository unavailable
- Insufficient disk space
**Solution:**
- Verify the cluster has outbound internet access
- Try building on a different cluster
- Check cluster storage availability
##### Issue: Advanced Options not taking effect
**Possible causes:**
- Service is using TCP or HTTPS Passthrough mode (Advanced Options only work with HTTP/HTTPS Termination modes)
- Configuration hasn't been reloaded
**Solution:**
- Verify the service mode is "HTTP" or "HTTPS Termination"
- Click "Reload Config" to apply the latest configuration
- Check the service card to confirm Advanced Options are saved
##### Issue: Legitimate users getting 429 Too Many Requests
**Possible causes:**
- Rate limit set too low
- Multiple users sharing one IP address (corporate NAT, VPN)
- Single page making many requests (heavy JavaScript, many images)
**Solution:**
- Increase the "Max requests per 10 seconds" value
- Consider the expected request pattern: a page with 20 images generates 21 requests
- Monitor typical usage before setting strict limits
##### Issue: Blocked path is needed for legitimate access
**Solution:**
- Edit the service and remove the path from the "Block Paths" field
- Be more specific with blocked paths (e.g., use `/.git` instead of `/.`)
- Consider using path-based routing to a separate service with different security settings
##### Issue: HSTS causing problems after disabling HTTPS
**Problem:** After enabling HSTS, browsers remember to use HTTPS. If you later disable HTTPS, users cannot access the site.
**Solution:**
- For affected users: Clear browser cache and HSTS data for the domain
- In Chrome: Visit `chrome://net-internals/#hsts` and delete the domain
- In Firefox: Clear "Site Preferences" or all history
- Prevention: Use short HSTS duration (1 day) for testing before committing to longer values
##### Issue: Compression not reducing response sizes
**Possible causes:**
- Content is already compressed (images, videos, PDFs)
- Responses are too small to benefit from compression
- Client doesn't advertise gzip support
**Solution:**
- Compression only works for text-based content (HTML, CSS, JS, JSON, XML)
- Verify the client sends `Accept-Encoding: gzip` header
- Check response headers for `Content-Encoding: gzip`
### 10. Glossary
|
Term |
Definition |
|
**VIP (Virtual IP)** |
A floating IP address that clients connect to. Managed by OVN and routes to HAProxy instances. |
|
**Backend** |
A server that receives traffic from HAProxy. Also called "upstream server" or "real server". |
|
**Frontend** |
The HAProxy component that accepts client connections (VIP + port + rules). |
|
**SNI (Server Name Indication)** |
A TLS extension that allows clients to specify the hostname they're connecting to, enabling multiple HTTPS sites on one IP. |
|
**SSL Termination** |
The process of decrypting SSL/TLS traffic at the load balancer, allowing it to inspect and modify HTTP content. |
|
**SSL Passthrough** |
Forwarding encrypted traffic directly to backends without decryption at the load balancer. |
|
**Health Check** |
Periodic test to verify a backend server is functioning correctly. |
|
**Sticky Session** |
A mechanism ensuring requests from the same client go to the same backend server. |
|
**Round Robin** |
A load balancing algorithm that distributes requests evenly in sequence. |
|
**OVN (Open Virtual Network)** |
Software-defined networking system used by INCUS for network virtualization. |
|
**Layer 4** |
Transport layer (TCP/UDP) - load balancing based on IP and port. |
|
**Layer 7** |
Application layer (HTTP) - load balancing based on content, headers, URLs. |
|
**HSTS** |
HTTP Strict Transport Security - a security header that tells browsers to only connect via HTTPS. |
|
**Rate Limiting** |
Restricting the number of requests a client can make within a time period to prevent abuse. |
|
**X-Frame-Options** |
HTTP header that controls whether a page can be displayed in an iframe, preventing clickjacking attacks. |
|
**X-Content-Type-Options** |
HTTP header that prevents browsers from MIME-sniffing a response away from the declared content type. |
|
**X-Forwarded-For** |
HTTP header containing the original client IP address when requests pass through proxies. |
|
**Gzip Compression** |
A compression algorithm that reduces the size of text-based HTTP responses to save bandwidth. |
|
**429 Too Many Requests** |
HTTP status code returned when a client exceeds the configured rate limit. |
|
**403 Forbidden** |
HTTP status code returned when access to a resource is denied (e.g., blocked path or user agent). |
**Need more help?** Contact your AETHER administrator or refer to the official HAProxy documentation at haproxy.org for advanced configuration options.
Reference in New Issue View Git Blame Copy Permalink