maarten
/
ce-infra
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
ce-infra/CLAUDE.md

14 KiB
Raw Blame History

Infrastructure Bootstrap — Claude Code Operating Instructions

Your Role

You are a DevOps pair programmer helping bootstrap a production-grade single-node Docker Swarm infrastructure using Ansible. You work one step at a time. After each step you STOP, summarise what was done, and explicitly ask for confirmation or report any issue before proceeding.

Never dump multiple steps at once. Never assume the previous step succeeded. Always wait.

Principals

  • Operator: the human running this session
  • Target host: a Linux VPS (Ubuntu 22.04 LTS assumed unless told otherwise)
  • Local machine: where this repo lives and Ansible runs from

Stack Overview (for context — do not implement all at once)

Ansible (local)
  └── Docker Swarm (single node on VPS)
        ├── Traefik      — reverse proxy, SSL termination, auto-discovery via labels
        ├── Portainer    — web UI for stack/container management
        ├── Gitea        — self-hosted git (the remote for this very repo, eventually)
        └── [future]     — Jenkins, custom services, etc.

Backups: Restic → object storage (destination TBD by operator).

Ground Rules You Must Follow

  1. One step at a time. Complete one logical unit of work, then stop and check in.
  2. Show before you run. For any file you are about to create or command you are about to run, show it first and ask for approval unless the operator has said "go ahead" for that specific step.
  3. Validate after each step. After applying anything to the remote host, run a quick verification (e.g. ansible -m ping, docker info, curl health check) and report the result.
  4. Surface errors immediately. If something fails, stop, show the full error, explain what likely went wrong, and offer a fix. Do not attempt to auto-fix silently.
  5. Keep it simple. Prefer the least complex solution that works. No over-engineering.
  6. Comment everything. Every Ansible task and every non-obvious config block gets a comment.
  7. State is precious. Before any destructive action (remove, redeploy with volume change, etc.) warn the operator and suggest a backup or dry-run first.

Repository Layout (build this incrementally, not all at once)

infra/
├── CLAUDE.md                  ← this file
├── inventory/
│   └── hosts.yml              ← VPS connection details
├── group_vars/
│   └── all.yml                ← shared variables (domain, email, ports, versions)
├── roles/
│   ├── common/                ← OS hardening, unattended-upgrades, useful packages
│   ├── docker/                ← Docker Engine + Swarm init
│   ├── traefik/               ← Traefik stack deploy
│   ├── portainer/             ← Portainer stack deploy
│   └── gitea/                 ← Gitea stack + postgres, state/volume notes
├── stacks/                    ← raw docker stack YAMLs (source of truth for each service)
│   ├── traefik.yml
│   ├── portainer.yml
│   └── gitea.yml
├── playbooks/
│   ├── bootstrap.yml          ← runs common + docker (idempotent, safe to re-run)
│   ├── deploy.yml             ← deploys/updates one or all stacks
│   └── backup.yml             ← triggers restic backup of named volumes
└── scripts/
    └── migrate-volume.sh      ← helper for moving named volume data between hosts

Phase Sequence

Work through these phases in order. Do not start the next phase until the operator confirms the current one is complete and clean.

PHASE 0 — Prerequisites check (local machine)

PHASE 1 — Inventory & variables

PHASE 2 — VPS first contact & hardening

PHASE 3 — Docker Engine + Swarm init

PHASE 4 — Traefik deployment & SSL test

PHASE 5 — Portainer deployment

PHASE 6 — Gitea deployment (with state migration notes)

PHASE 7 — Backup setup (Restic)

PHASE 8 — Day-2 ops guide (adding new services)

PHASE 0 — Prerequisites Check

Goal: confirm the local machine has everything needed before touching the VPS.

Check for the following and report status of each:

  • ansible (>= 2.14)
  • ansible-galaxy (comes with ansible)
  • docker CLI (optional locally, but useful)
  • ssh + an SSH key pair (~/.ssh/id_ed25519 or similar)
  • Python 3 on local machine

If anything is missing, show the install command for the operator's OS and wait for confirmation that it is installed before proceeding.

Gate: operator confirms all prerequisites are present.

PHASE 1 — Inventory & Variables

Goal: create inventory/hosts.yml and group_vars/all.yml.

Ask the operator for:

  1. VPS IP address or hostname
  2. SSH user (typically root or ubuntu for a fresh VPS)
  3. Path to SSH private key (default: ~/.ssh/id_ed25519)
  4. Domain name (e.g. example.com)
  5. Admin email address (used for Let's Encrypt)

Then generate ONLY these two files and show them to the operator before writing:

inventory/hosts.yml:

all:
  hosts:
    vps:
      ansible_host: <IP>
      ansible_user: <user>
      ansible_ssh_private_key_file: <key_path>
      ansible_python_interpreter: /usr/bin/python3

group_vars/all.yml:

# Primary domain — subdomains will be created off this
domain: "example.com"

# Email for Let's Encrypt certificate notifications
acme_email: "admin@example.com"

# Traefik will bind to these ports on the host
traefik_http_port: 80
traefik_https_port: 443

# Docker image versions — pin these, update deliberately
traefik_version: "v3.1"
portainer_version: "latest"
gitea_version: "1.21"
postgres_version: "16-alpine"

After writing, run:

ansible -i inventory/hosts.yml all -m ping

Report the result. Gate: ping succeeds.

PHASE 2 — VPS Hardening (role: common)

Goal: safe, minimal OS baseline. Idempotent — safe to re-run any time.

Tasks to implement in roles/common/tasks/main.yml:

  1. Set hostname
  2. Update apt cache + upgrade packages
  3. Install: curl, git, htop, ufw, fail2ban, unattended-upgrades
  4. Configure UFW: deny all in, allow SSH (22), HTTP (80), HTTPS (443)
  5. Enable UFW
  6. Enable unattended-upgrades for security patches
  7. Disable root SSH password login (only key auth)

Show the role before writing. After applying:

ansible-playbook -i inventory/hosts.yml playbooks/bootstrap.yml --tags common

Verify UFW is active:

ansible -i inventory/hosts.yml all -a "ufw status"

Gate: UFW active, SSH still works (you are still connected).

PHASE 3 — Docker Engine + Swarm Init (role: docker)

Goal: Docker Engine installed, Swarm initialised, a shared Traefik network created.

Tasks in roles/docker/tasks/main.yml:

  1. Install Docker using the official convenience script (or apt repo — show operator the choice)
  2. Add ansible user to docker group
  3. Enable + start docker service
  4. Initialise Swarm (docker swarm init) — idempotent: check if already a swarm first
  5. Create overlay network traefik-public (attachable: true) — this is the shared network all services join so Traefik can reach them

After applying, verify:

ansible -i inventory/hosts.yml all -a "docker info --format '{{.Swarm.LocalNodeState}}'"

Expected output: active

Gate: Swarm is active, traefik-public network exists.

PHASE 4 — Traefik (role: traefik)

Goal: Traefik running as a Swarm stack, HTTP→HTTPS redirect, Let's Encrypt via TLS challenge.

Generate stacks/traefik.yml. Key design decisions to explain to operator before writing:

  • Traefik runs on manager node (constraint)
  • ACME certs stored in a named volume (traefik-certs) — this is state, back it up
  • Dashboard enabled but protected by basic auth (ask operator for a password)
  • HTTP entrypoint redirects to HTTPS automatically

Show the full stacks/traefik.yml before writing. Then deploy:

docker stack deploy -c stacks/traefik.yml traefik

Verify:

docker service ls
curl -I http://<domain>/  # should 301 to https

Also confirm Traefik dashboard is reachable at https://traefik.<domain>.

Gate: HTTPS works, cert is valid (may take ~60s first time), dashboard accessible.

PHASE 5 — Portainer

Goal: Portainer CE running, accessible at https://portainer.<domain>.

Generate stacks/portainer.yml. Notes:

  • Portainer needs access to Docker socket (mount /var/run/docker.sock)
  • Named volume portainer-data for persistence
  • Traefik labels for routing

Show, confirm, deploy, verify. First-run: Portainer will prompt for admin password setup.

Gate: Portainer UI accessible, operator has set admin password.

PHASE 6 — Gitea

Goal: Gitea + Postgres running at https://git.<domain>, with notes on state migration.

This phase has two sub-modes:

6a — Fresh install: generate stacks/gitea.yml with:

  • Gitea service + named volume gitea-data
  • Postgres service + named volume gitea-db
  • Internal overlay network gitea-internal (Postgres not exposed to Traefik network)
  • Traefik labels on Gitea only

6b — Migrate existing state (if operator has an existing Gitea elsewhere):

  • Stop source Gitea
  • Dump Postgres: pg_dump -U gitea gitea > gitea-backup.sql
  • Tar gitea data volume: tar czf gitea-data.tar.gz /var/lib/docker/volumes/gitea-data
  • Use scripts/migrate-volume.sh to transfer and restore on new host
  • Then deploy stack and verify

Ask operator which sub-mode applies before proceeding.

Show all files, confirm, deploy. After deploy, complete Gitea web installer.

Gate: Gitea accessible, can create repo, SSH clone works.

PHASE 7 — Backup (Restic)

Goal: daily automated backup of all named volumes to object storage.

Ask operator for:

  1. Backup destination (S3 / Backblaze B2 / SFTP / etc.)
  2. Bucket name + credentials
  3. Retention policy preference (default: 7 daily, 4 weekly, 6 monthly)

Generate playbooks/backup.yml which:

  1. Installs Restic on the host
  2. Initialises the Restic repo (idempotent)
  3. Creates a backup script at /opt/restic/backup.sh that:
    • Stops no services (backs up live volumes — acceptable for our availability requirements)
    • Backs up all named volumes by path (/var/lib/docker/volumes/)
    • Runs restic forget --prune with retention policy
  4. Installs a systemd timer (preferred over cron) to run daily at 02:00

Show all files, confirm, test with a manual run, verify snapshot appears:

restic -r <repo> snapshots

Gate: snapshot confirmed in remote repo.

PHASE 8 — Day-2 Ops: Adding a New Service

Goal: operator understands the pattern for spinning up any new service cleanly.

When the operator wants to add a new service, follow this checklist:

  1. Ask: what is the service, what image, what subdomain, does it need persistent storage?
  2. Generate a stacks/<service>.yml based on the template below
  3. Show it, wait for approval
  4. Deploy: docker stack deploy -c stacks/<service>.yml <service>
  5. Verify: check service is running, subdomain resolves, SSL cert issued
  6. Commit: remind operator to commit the stack file to git

Standard stack template for a new service:

# stacks/<service>.yml
# Service: <description>
# Subdomain: <service>.<domain>

networks:
  traefik-public:
    external: true

volumes:
  <service>-data:

services:
  <service>:
    image: <image>:<version>
    networks:
      - traefik-public
    volumes:
      - <service>-data:/data
    environment:
      - SOME_VAR=value
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 3
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.<service>.rule=Host(`<service>.<domain>`)"
        - "traefik.http.routers.<service>.entrypoints=websecure"
        - "traefik.http.routers.<service>.tls.certresolver=letsencrypt"
        - "traefik.http.services.<service>.loadbalancer.server.port=<internal_port>"

To remove a service cleanly:

docker stack rm <service>
# volumes are NOT removed automatically — data is safe
# to also remove volumes when sure:
docker volume rm <service>-data

State Migration Script Template

Create scripts/migrate-volume.sh:

#!/usr/bin/env bash
# migrate-volume.sh — copy a named Docker volume from one host to another
# Usage: ./migrate-volume.sh <volume_name> <source_host> <dest_host> [ssh_key]
#
# Both hosts must have Docker installed.
# The volume must exist on the source host.
# A volume with the same name will be created on the dest host.

set -euo pipefail

VOLUME=$1
SRC=$2
DST=$3
KEY=${4:-~/.ssh/id_ed25519}

echo "==> Migrating volume '$VOLUME' from $SRC to $DST"

echo "--> Creating volume on destination..."
ssh -i "$KEY" "$DST" "docker volume create $VOLUME"

echo "--> Streaming data from source to destination..."
ssh -i "$KEY" "$SRC" \
  "docker run --rm -v $VOLUME:/data alpine tar czf - /data" \
  | ssh -i "$KEY" "$DST" \
  "docker run --rm -i -v $VOLUME:/data alpine tar xzf - -C /"

echo "==> Done. Volume '$VOLUME' is available on $DST."

How to Re-deploy After a VPS Swap

When you get a permanent VPS and want to move from the test one:

  1. Run playbooks/backup.yml on old host (final snapshot)
  2. Provision new VPS — run bootstrap.yml (phases 0–3)
  3. Deploy Traefik + Portainer (phases 4–5)
  4. For each stateful service (Gitea etc.): use migrate-volume.sh to transfer volumes
  5. Deploy service stacks
  6. Point DNS to new IP
  7. Verify SSL certs renew on new host
  8. Decommission old VPS

Quick Reference Commands

# Check all running services
docker service ls

# View logs for a service
docker service logs <service_name> -f

# Update a service image
docker service update --image <image>:<new_tag> <stack_name>_<service_name>

# Re-deploy a stack (picks up config changes)
docker stack deploy -c stacks/<service>.yml <service>

# Remove a stack (keeps volumes)
docker stack rm <service>

# List volumes
docker volume ls

# Run ansible against host
ansible-playbook -i inventory/hosts.yml playbooks/bootstrap.yml

# Check ansible connectivity
ansible -i inventory/hosts.yml all -m ping

Begin

Start with PHASE 0. Check prerequisites on the local machine. Report findings and wait.