Hosting Archestack on AKS

This guide installs an Archestack instance on Azure Kubernetes Service (AKS) and then keeps it up to date by itself. The Helm chart runs the API as several identical replicas, the builder and the published app, and Caddy as the edge with automatic certificates. PostgreSQL runs as Azure Database for PostgreSQL Flexible Server, and the images come from a container registry of your own. A pipeline looks up the newest Archestack release every night, takes it to development and test, and takes that same version on to acceptance and production once someone approves it. On a single server, Hosting with Docker is the simpler way.

Already have a cluster and want Archestack on it today, under a path of a hostname you already serve? Archestack on Kubernetes, the short way does that with five files and one pipeline, and is the place to start.

What runs where

                  LoadBalancer, static public IP (80, 443)
                                 |
                               Caddy ---- asks the API before every new certificate
         +-------------+---------+---------+------------------------------+
   editor.<zone>   api.<zone>    metrics.<zone>    <workspace>.<zone>, companies' own domains
         |             |              |                          |
   web-dashboard     api (N)       grafana                     web-app
                       |
     Azure Database for PostgreSQL 16, Flexible Server, private to the network
     Azure Files: /files and /backups, one share every API replica mounts
     omnicore-api and omnicore-web, pulled from your own registry (ACR)

• Caddy is the edge, not an ingress controller. Workspace addresses and companies' own domains get a certificate the first time someone visits them, and Caddy asks the API whether a hostname belongs to a workspace before it requests one. cert-manager cannot issue certificates on demand, so an ingress controller would lose those addresses. Without them, Archestack can also sit under a path of a hostname the cluster already serves.
• The visitor's address is kept (externalTrafficPolicy: Local). The rate limiter counts anonymous callers by address, and without it every visitor would share a node's address.
• Files and backups are one share on Azure Files that every API replica mounts, so a logo or a backup written through one replica is there on all of them. The NuGet packages scripts use are the exception: each pod keeps its own on its node, for as long as the pod lasts (api.nugetCache).
• The API creates its own databases and migrates them when it starts, so an empty server is all it needs. There is no separate migration job.

Versions and tags

Every Archestack release has a version number, major.minor.build, such as 1.4.23. The build number counts up with each release and starts again at 1 when the major or minor version changes, and a number can be missing when a release build did not finish. Each release publishes two images on Docker Hub, nickpeelman/omnicore-api and nickpeelman/omnicore-web, each with three tags: the version, the commit it was built from, and latest.

Deploy the version number, never latest:

• Nothing rolls out. An upgrade from latest to latest leaves the Deployment unchanged, so Kubernetes keeps the pods it has.
• Replicas drift apart. The chart pulls an image only when the node does not have it yet. A node that pulled latest last week keeps running last week's version, a new node runs this week's, and one instance serves two versions.
• There is no way back. The version before has no name to go back to.

The pipeline below therefore turns "newest" into a number once, at the start of a run, and every environment gets that number. It can also follow one line: 1.4 takes the newest 1.4.x and never moves to 1.5 by itself. A new major or minor version is where the larger changes land, and the VS Code extension warns when it connects to an API of another major or minor version.

Before you start

• An Azure subscription, and the az CLI, kubectl and Helm 3 on your machine.
• A project on Azure DevOps, GitHub or GitLab for the pipeline, and a repository there made from the starter kit.
• The Helm chart: the infra/helm/archestack folder of the Archestack repository, copied into that repository at the same path.
• A DNS zone you can add records to, or a hostname a cluster already serves through its ingress controller (way B).

The starter kit

infra/helm/archestack/       the chart, copied from the Archestack repository
setup/                       run once per environment, by a person
  1-cluster.sh               registry, network, Key Vault, database, cluster
  2-address.sh               way A: the public address for DNS
  3-pipeline.sh              the SMTP password, the namespace, the pipeline's identity
aks/
  settings.env               fill in: prefix, region, where the pipeline runs
  environment.sh             every Azure name, made from settings.env
  resolve-version.sh         newest, 1.4 or 1.4.23, made one version number
  deploy.sh                  import into your registry, then helm upgrade
  azure-pipelines.yml        the pipeline, on Azure DevOps
  environments/
    dev/ tst/ acc/ prd/      fill in: values.yaml, and names.env for a cluster you have
.github/workflows/           the pipeline, on GitHub Actions
.gitlab-ci.yml               the pipeline, on GitLab CI

Keep the pipeline of your CI system and delete the other two. grep -rn CHANGE-ME . lists every value still to fill in, and deploy.sh refuses an environment that still has one. The three setup scripts then make each environment in Azure, run by a person once:

# Once per environment, after az login as someone who may create resources and assign roles
ENVIRONMENT=prd bash setup/1-cluster.sh       # registry, network, Key Vault, database, cluster
ENVIRONMENT=prd bash setup/2-address.sh       # way A only: the public address, then your DNS
ENVIRONMENT=prd bash setup/3-pipeline.sh      # asks the SMTP password; the pipeline's identity

Each script makes only what is missing, so one that stopped halfway can simply be run again, and each ends by saying what comes next.

1. Fill in the settings

aks/settings.env holds what every environment shares:

# aks/settings.env: fill in once. No passwords here: the setup asks for them and keeps them in Key Vault.
PREFIX=CHANGE-ME             # 2-10 lowercase letters or digits, unique to you, such as acme
LOCATION=westeurope          # the Azure region for every environment
PIPELINE=CHANGE-ME           # where the pipeline runs: azure-devops, github or gitlab
REPO=                        # github: owner/repo; gitlab: group/project; azure-devops: leave empty
GITLAB_URL=https://gitlab.com

Every Azure name follows from it in aks/environment.sh, which the setup and deploy.sh both read, so they always agree. Names Azure needs unique worldwide (the registry, the vault and the database server) end in your prefix:

# aks/environment.sh: one environment's Azure names, made from settings.env, and how to sign in to
# its cluster. The setup scripts and deploy.sh both read it, so they always agree.
# shellcheck source=settings.env
source "$(dirname "${BASH_SOURCE[0]}")/settings.env"
: "${ENVIRONMENT:?set ENVIRONMENT to dev, tst, acc or prd}"
[[ $PREFIX =~ ^[a-z0-9]{2,10}$ ]] || { echo "Fill in PREFIX in aks/settings.env: 2-10 lowercase letters or digits" >&2; exit 1; }

RG=rg-archestack-$ENVIRONMENT
AKS_RG=$RG
AKS=aks-archestack-$ENVIRONMENT
VNET_RG=$RG
VNET=vnet-archestack-$ENVIRONMENT
PG_SUBNET=10.20.16.0/24
PIP=pip-archestack-$ENVIRONMENT
IDENTITY=id-archestack-$ENVIRONMENT
PG=pg-$PREFIX-$ENVIRONMENT          # <name>.postgres.database.azure.com, unique across Azure
KV=kv-$PREFIX-$ENVIRONMENT          # unique across Azure, at most 24 characters
ACR=acr$PREFIX                      # one registry for every environment, unique across Azure
ACR_RG=rg-archestack-shared
NAMESPACE=archestack

# A cluster you already have, such as one whose ingress already serves the hostname Archestack
# goes under: aks/environments/<env>/names.env changes the names above (see names.env.example).
OVERRIDES="$(dirname "${BASH_SOURCE[0]}")/environments/$ENVIRONMENT/names.env"
if [ -f "$OVERRIDES" ]; then source "$OVERRIDES"; fi

# Signs kubectl and helm in to the cluster as whoever az is signed in as. A cluster with Entra ID
# sign-in needs kubelogin for that, installed into ~/.local/bin when it is missing.
kube_login() {
  az aks get-credentials -g "$AKS_RG" -n "$AKS" --overwrite-existing -o none
  if [ "$(az aks show -g "$AKS_RG" -n "$AKS" --query 'aadProfile != null' -o tsv)" = true ]; then
    if ! command -v kubelogin > /dev/null; then
      mkdir -p "$HOME/.local/bin"
      az aks install-cli --install-location "$HOME/.local/bin/kubectl" \
        --kubelogin-install-location "$HOME/.local/bin/kubelogin" -o none
      export PATH="$HOME/.local/bin:$PATH"
    fi
    kubelogin convert-kubeconfig -l azurecli
  fi
}

Each environment's values.yaml holds its chart values and one of two ways in: hostnames of its own (A), or a path of a hostname the cluster already serves (B). Keep one, delete the other.

# aks/environments/prd/values.yaml: the chart values of this environment. The chart's own values.yaml
# lists every value. The images, the database server and the public address come from deploy.sh,
# the passwords from Key Vault.
#
# Two ways in. Keep one, delete the other, and fill in every CHANGE-ME that is left:
#   A. Hostnames of its own (editor., api., metrics., and <workspace>. per workspace) behind Caddy,
#      with certificates it gets by itself. Needs DNS records and setup/2-address.sh.
#   B. A path of a hostname the cluster's ingress already serves, such as https://apps.acme.com/archestack/.
#      No new DNS record, address or certificate. No workspace addresses, own domains or Grafana.

# --- A. Hostnames of its own ---
hosts:
  dashboard: editor.CHANGE-ME          # editor.<your zone>, such as editor.archestack.acme.com
  api: api.CHANGE-ME                   # api.<your zone>
  grafana: metrics.CHANGE-ME           # metrics.<your zone>
caddy:
  acmeEmail: CHANGE-ME                 # Let's Encrypt writes here about certificates
  persistence:
    storageClass: managed-csi

# --- B. A path of a hostname the cluster already serves ---
ingress:
  enabled: true
  className: nginx                     # the ingress class that serves the hostname
  host: CHANGE-ME                      # such as apps.acme.com
  basePath: /archestack                # the builder at https://<host>/archestack/
  tls:
    - secretName: CHANGE-ME            # the hostname's certificate, as a Secret in this namespace
      hosts: [CHANGE-ME]               # the host again

# --- Both ---
api:
  replicas: 3
  # A: your zone, for workspaces at <label>.<zone> and companies' own domains, or empty for neither.
  # B: empty.
  tenantHostSuffix: CHANGE-ME
  persistence:
    files: { storageClass: azurefile-csi, accessMode: ReadWriteMany, size: 100Gi }
    backups: { storageClass: azurefile-csi, accessMode: ReadWriteMany, size: 100Gi }

postgres:
  username: archestack
  sslMode: Require

# The mail server; its password is in Key Vault. Leave all three empty when the instance sends no mail.
smtp:
  host: CHANGE-ME
  username: CHANGE-ME
  from: CHANGE-ME

observability:
  enabled: true                        # B: Grafana is not routed; reach it with kubectl port-forward
  instance: prd
  storageClass: managed-csi

• api.replicas: any number, with no sticky sessions. Hosting infrastructure lists what several replicas ask of the network around them.
• api.tenantHostSuffix: turns on an address per workspace and companies' own domains, with way A only.
• api.persistence: has to be ReadWriteMany as soon as replicas run on more than one node.
• observability.enabled: Grafana, Prometheus, Tempo and Loki inside the cluster. Their host and container panels stay empty on AKS; Container Insights covers the nodes and containers.

Under a path of a hostname you already have

Way B puts Archestack under a path of a hostname the cluster's ingress controller already serves, beside the applications already on it:

https://apps.acme.com/                       what is already there
https://apps.acme.com/archestack/            the builder
https://apps.acme.com/archestack/app/<page>  the published app
https://apps.acme.com/archestack/api         the API (and /hubs, live updates)

It needs no new DNS record, no public address and no certificate of its own, and people keep using the address they already know. The chart leaves Caddy out and renders one Ingress for the three paths; the builder and the API are told their prefix, so every link they write carries it.

# --- B. A path of a hostname the cluster already serves ---
ingress:
  enabled: true
  className: nginx                     # the ingress class that serves the hostname
  host: CHANGE-ME                      # such as apps.acme.com
  basePath: /archestack                # the builder at https://<host>/archestack/
  tls:
    - secretName: CHANGE-ME            # the hostname's certificate, as a Secret in this namespace
      hosts: [CHANGE-ME]               # the host again

• A cluster you already have. The hostname lives on a cluster, usually one that was there before Archestack. Copy names.env.example to aks/environments/<env>/names.env and the setup leaves that cluster as it is: it allows the cluster to pull from the registry and adds a subnet for the database to its network. Skip setup/2-address.sh.
• className is the ingress class that serves the hostname. The chart sets ingress-nginx's annotations for large uploads and long-lived connections; another controller needs its own equivalents.
• The certificate has to be a Secret in the archestack namespace, because an Ingress only uses Secrets in its own namespace. Copy the hostname's certificate there, or have cert-manager issue it there.
• Do not rewrite the path (no rewrite-target). The API needs to see its prefix to write its own addresses.
• Keep sign-in proxies off it. One that replaces the Authorization header breaks every API call: the published app, Data API keys, deploy tokens, VS Code and MCP all bring their own.
• The hostname is shared. Sign-in tokens live in the browser's storage for that address, which every application on it can read, so a script injected into any of them reaches Archestack sessions. Two Archestack instances on one hostname sign each other's users out.
• Not available this way: workspace addresses and companies' own domains, which need Caddy's certificates on demand, and Grafana, which the Ingress does not route. Reach Grafana with kubectl port-forward.

# A cluster you already have, for one environment: copy this file to aks/environments/<env>/names.env
# and fill it in. The setup then leaves the cluster as it is, allows it to pull from the registry, and
# adds a subnet for the database to the cluster's network. Everything else is made as usual.

# The cluster, such as the one whose ingress already serves the hostname Archestack goes under.
AKS_RG=CHANGE-ME
AKS=CHANGE-ME

# Its network, which has to be one of your own (a cluster made with --vnet-subnet-id), and a free
# address range in it for the database's subnet.
VNET_RG=CHANGE-ME
VNET=CHANGE-ME
PG_SUBNET=CHANGE-ME                 # such as 10.30.16.0/24

2. Registry, cluster and database

#!/usr/bin/env bash
# setup/1-cluster.sh: one environment's registry, network, Key Vault, database and cluster.
# Each part is made only when it is missing, so a run that stopped halfway can simply be run again,
# and a cluster you already have (aks/environments/<env>/names.env) is left as it is.
#   ENVIRONMENT=prd bash setup/1-cluster.sh
# Smaller, for dev and test:
#   NODE_COUNT=2 ENVIRONMENT=dev bash setup/1-cluster.sh
source "$(dirname "$0")/common.sh"
NODE_COUNT=${NODE_COUNT:-3}
NODE_SIZE=${NODE_SIZE:-Standard_D4s_v5}
PG_SKU=${PG_SKU:-Standard_D2ds_v5}
exists() { "$@" -o none 2>/dev/null; }

# One registry for every environment: a version is imported once and promoted as it is.
az group create -n "$ACR_RG" -l "$LOCATION" -o none
exists az acr show -n "$ACR" || az acr create -g "$ACR_RG" -n "$ACR" --sku Standard -o none
ACR_ID=$(az acr show -n "$ACR" --query id -o tsv)

# The network: a subnet for the cluster, and one delegated to the database.
az group create -n "$RG" -l "$LOCATION" -o none
exists az network vnet show -g "$VNET_RG" -n "$VNET" || az network vnet create -g "$VNET_RG" -n "$VNET" \
  --address-prefixes 10.20.0.0/16 --subnet-name aks --subnet-prefixes 10.20.0.0/20 -o none
exists az network vnet subnet show -g "$VNET_RG" --vnet-name "$VNET" -n archestack-postgres \
  || az network vnet subnet create -g "$VNET_RG" --vnet-name "$VNET" -n archestack-postgres \
    --address-prefixes "$PG_SUBNET" --delegations Microsoft.DBforPostgreSQL/flexibleServers -o none

# The vault, and the right to write to it: a vault with RBAC gives its creator no access to secrets.
exists az keyvault show -n "$KV" || az keyvault create -g "$RG" -n "$KV" -l "$LOCATION" --enable-rbac-authorization true -o none
az role assignment create --assignee "$(az ad signed-in-user show --query id -o tsv)" \
  --role "Key Vault Secrets Officer" --scope "$(az keyvault show -n "$KV" --query id -o tsv)" -o none

# PostgreSQL 16, reachable from the network only. Its password is made here and goes straight
# into the vault, so nobody has to know it. No ';' in it: it goes into a connection string.
exists az postgres flexible-server show -g "$RG" -n "$PG" || {
  PG_PASSWORD=$(openssl rand -base64 36 | tr -d '/+=;')
  kv_set archestack-postgres-password "$PG_PASSWORD"
  az postgres flexible-server create -g "$RG" -n "$PG" -l "$LOCATION" --version 16 \
    --tier GeneralPurpose --sku-name "$PG_SKU" --storage-size 128 \
    --admin-user archestack --admin-password "$PG_PASSWORD" \
    --subnet "$(az network vnet subnet show -g "$VNET_RG" --vnet-name "$VNET" -n archestack-postgres --query id -o tsv)" \
    --private-dns-zone "$PG.private.postgres.database.azure.com" --yes -o none
}

# The cluster, allowed to pull from the registry.
if exists az aks show -g "$AKS_RG" -n "$AKS"; then
  az aks update -g "$AKS_RG" -n "$AKS" --attach-acr "$ACR_ID" -o none
else
  az aks create -g "$AKS_RG" -n "$AKS" --tier standard \
    --node-count "$NODE_COUNT" --node-vm-size "$NODE_SIZE" \
    --network-plugin azure --network-plugin-mode overlay --pod-cidr 10.244.0.0/16 \
    --vnet-subnet-id "$(az network vnet subnet show -g "$VNET_RG" --vnet-name "$VNET" -n aks --query id -o tsv)" \
    --enable-managed-identity --generate-ssh-keys --attach-acr "$ACR_ID" -o none
fi

echo "$ENVIRONMENT: $AKS, $PG and $KV are ready."
echo "Next: ENVIRONMENT=$ENVIRONMENT bash setup/2-address.sh, or with a path of a hostname the"
echo "cluster already serves (way B in values.yaml): ENVIRONMENT=$ENVIRONMENT bash setup/3-pipeline.sh"

One registry serves every environment, so a version is imported once and production runs the very image acceptance ran. The database password is made by the script and goes straight into Key Vault: nobody needs to know it. Capacity and sizing has the measured figures behind the sizes, which are a starting point for production.

Every workspace owns two databases. An idle one holds no connections, but a busy server needs room: the Burstable tier's limit of 50 connections is too low. The API needs no extensions and no superuser, only an admin user that can create databases.

3. Public address and DNS

#!/usr/bin/env bash
# setup/2-address.sh: the environment's static public address, for Caddy and your DNS. Not needed
# with a path of a hostname the cluster already serves (way B in values.yaml).
#   ENVIRONMENT=prd bash setup/2-address.sh
source "$(dirname "$0")/common.sh"

# In the cluster's node resource group, because that is where its load balancer looks for it by name.
NODE_RG=$(az aks show -g "$AKS_RG" -n "$AKS" --query nodeResourceGroup -o tsv)
az network public-ip create -g "$NODE_RG" -n "$PIP" --sku Standard --allocation-method Static -o none
IP=$(az network public-ip show -g "$NODE_RG" -n "$PIP" --query ipAddress -o tsv)

echo "Point these DNS records at $IP, under the zone in aks/environments/$ENVIRONMENT/values.yaml:"
echo "  editor  api  metrics  *"
echo "Next: ENVIRONMENT=$ENVIRONMENT bash setup/3-pipeline.sh"

Way A only. Point editor, api and metrics under your zone at the address, plus a wildcard * for the workspaces' own addresses. Caddy gets its certificates from Let's Encrypt, so ports 80 and 443 have to be reachable from the internet.

4. Passwords and the pipeline's identity

#!/usr/bin/env bash
# setup/3-pipeline.sh: the SMTP password, the namespace, and the identity the pipeline deploys
# this environment as.
#   ENVIRONMENT=prd bash setup/3-pipeline.sh
source "$(dirname "$0")/common.sh"
case "$PIPELINE" in
  azure-devops) ;;
  github | gitlab) [ -n "$REPO" ] || { echo "Fill in REPO in aks/settings.env" >&2; exit 1; } ;;
  *) echo "Fill in PIPELINE in aks/settings.env: azure-devops, github or gitlab" >&2; exit 1 ;;
esac

# Asked for here, so it is never written to a file. Enter nothing to keep what the vault has,
# or when the instance sends no mail.
read -rsp "SMTP password for $ENVIRONMENT (Enter to skip): " SMTP_PASSWORD; echo
if [ -n "$SMTP_PASSWORD" ]; then kv_set archestack-smtp-password "$SMTP_PASSWORD"; fi

# The pipeline's identity, one per environment, and what it may do: sign in to the cluster,
# read the passwords, and import into the registry.
az identity create -g "$RG" -n "$IDENTITY" -l "$LOCATION" -o none
CLIENT_ID=$(az identity show -g "$RG" -n "$IDENTITY" --query clientId -o tsv)
PRINCIPAL_ID=$(az identity show -g "$RG" -n "$IDENTITY" --query principalId -o tsv)
AKS_ID=$(az aks show -g "$AKS_RG" -n "$AKS" --query id -o tsv)
grant() {
  az role assignment create --assignee-object-id "$PRINCIPAL_ID" --assignee-principal-type ServicePrincipal \
    --role "$1" --scope "$2" -o none
}
grant "Azure Kubernetes Service Cluster User Role" "$AKS_ID"
grant "Key Vault Secrets User" "$(az keyvault show -n "$KV" --query id -o tsv)"
grant Contributor "$(az acr show -n "$ACR" --query id -o tsv)"

# The namespace, made here by you, so the pipeline only ever needs rights inside it. A cluster
# with Entra ID sign-in checks those rights itself: Azure RBAC, or a Kubernetes RoleBinding.
kube_login
kubectl create namespace "$NAMESPACE" --dry-run=client -o yaml | kubectl apply -f - > /dev/null
ENTRA=$(az aks show -g "$AKS_RG" -n "$AKS" --query 'aadProfile != null' -o tsv)
AZURE_RBAC=$(az aks show -g "$AKS_RG" -n "$AKS" --query 'aadProfile.enableAzureRbac == `true`' -o tsv)
if [ "$AZURE_RBAC" = true ]; then
  grant "Azure Kubernetes Service RBAC Writer" "$AKS_ID/namespaces/$NAMESPACE"
elif [ "$ENTRA" = true ]; then
  kubectl create rolebinding archestack-pipeline -n "$NAMESPACE" --clusterrole=edit --user="$PRINCIPAL_ID" \
    --dry-run=client -o yaml | kubectl apply -f - > /dev/null
fi

# No stored secret: the CI system hands the pipeline a token that says who it is, and the
# identity trusts only tokens with this issuer and subject.
federate() {
  az identity federated-credential create -g "$RG" --identity-name "$IDENTITY" -n "$1" \
    --issuer "$2" --subject "$3" --audiences "$4" -o none
}
TENANT_ID=$(az account show --query tenantId -o tsv)
SUBSCRIPTION_ID=$(az account show --query id -o tsv)
case "$PIPELINE" in
  github)
    federate "github-$ENVIRONMENT" https://token.actions.githubusercontent.com \
      "repo:$REPO:environment:archestack-$ENVIRONMENT" api://AzureADTokenExchange
    echo "Done. In GitHub, create the environment archestack-$ENVIRONMENT, then run:"
    echo "  gh variable set AZURE_CLIENT_ID --env archestack-$ENVIRONMENT --body $CLIENT_ID --repo $REPO"
    echo "  gh variable set AZURE_TENANT_ID --body $TENANT_ID --repo $REPO"
    echo "  gh variable set AZURE_SUBSCRIPTION_ID --body $SUBSCRIPTION_ID --repo $REPO"
    ;;
  gitlab)
    federate "gitlab-$ENVIRONMENT" "$GITLAB_URL" \
      "project_path:$REPO:ref_type:branch:ref:main" "$GITLAB_URL"
    echo "Done. In GitLab, Settings, CI/CD, Variables, add:"
    echo "  AZURE_CLIENT_ID = $CLIENT_ID    (environment scope: archestack-$ENVIRONMENT)"
    echo "  AZURE_TENANT_ID = $TENANT_ID    (all environments)"
    echo "  AZURE_SUBSCRIPTION_ID = $SUBSCRIPTION_ID    (all environments)"
    ;;
  azure-devops)
    if [ -n "${AZDO_ISSUER:-}" ]; then
      federate "azure-devops-$ENVIRONMENT" "$AZDO_ISSUER" "$AZDO_SUBJECT" api://AzureADTokenExchange
      echo "Done. sc-archestack-$ENVIRONMENT can deploy $ENVIRONMENT; verify it in Azure DevOps."
    else
      echo "In Azure DevOps, add a service connection sc-archestack-$ENVIRONMENT: Azure Resource Manager,"
      echo "workload identity federation, set up manually, with this identity:"
      echo "  tenant $TENANT_ID, subscription $SUBSCRIPTION_ID, client id $CLIENT_ID"
      echo "It then shows an issuer and a subject identifier. Run this again with them:"
      echo "  AZDO_ISSUER='<issuer>' AZDO_SUBJECT='<subject identifier>' ENVIRONMENT=$ENVIRONMENT bash setup/3-pipeline.sh"
    fi
    ;;
esac

• The SMTP password is asked for and goes straight into Key Vault. An instance that sends no mail skips it.
• The pipeline signs in without a stored secret. Each environment has an identity of its own, which trusts only the token its CI system hands that environment's jobs. It may sign in to the cluster, read the passwords and import into the registry, nothing else.
• The namespace is made by you, so the pipeline only needs rights inside it. A cluster with Entra ID sign-in checks those rights itself, through Azure RBAC or a Kubernetes RoleBinding; the script gives whichever the cluster uses.
• The script ends with what to set in your CI system: GitHub variables, GitLab variables, or the Azure DevOps service connection.

5. The first install

The first install is the script the pipeline runs, started by hand. A first run of the pipeline does the same.

# From the repository, after az login as the person who ran the setup.
bash aks/resolve-version.sh                          # writes the newest version to ./version
VERSION=$(cat version) ENVIRONMENT=prd bash aks/deploy.sh
kubectl -n archestack get deploy -o wide             # every Deployment ready, and its image

# Grafana's password, generated on the first install and kept from then on
kubectl -n archestack get secret archestack-secrets -o jsonpath='{.data.admin-password}' | base64 -d

Open the builder straight away and create the first admin. Until someone does, whoever reaches it first becomes the admin. To leave no window at all, seed the first admin with api.initialAdmin (see the chart's values.yaml).

The newest release, automatically

resolve-version.sh asks the registry for every tag of both images and picks the version this run deploys:

#!/usr/bin/env bash
# aks/resolve-version.sh: turn "newest" into the version number this run deploys.
#   ARCHESTACK_VERSION=newest    the newest release
#   ARCHESTACK_VERSION=1.4       the newest 1.4.x
#   ARCHESTACK_VERSION=1.4.23    exactly that one
set -euo pipefail
export LC_ALL=C
SOURCE=${SOURCE_NAMESPACE:-nickpeelman}
WANT=${ARCHESTACK_VERSION:-newest}
OUT=${OUT_DIR:-.}

# Every major.minor.build tag of one image, from the registry API. The commit tags and
# "latest" do not match the pattern.
tags() {
  local repo="$SOURCE/$1" token
  token=$(curl -fsS "https://auth.docker.io/token?service=registry.docker.io&scope=repository:$repo:pull" \
    | sed -nE 's/.*"token":"([^"]+)".*/\1/p')
  curl -fsS -H "Authorization: Bearer $token" "https://registry-1.docker.io/v2/$repo/tags/list" \
    | grep -oE '"[0-9]+\.[0-9]+\.[0-9]+"' | tr -d '"' | sort -u
}

# A version counts only when both images carry it: a release build that stopped between
# the two pushes leaves one of them without it. Sorted as versions, so 1.0.10 beats 1.0.9.
versions=$(comm -12 <(tags omnicore-api) <(tags omnicore-web) | sort -V)

if [ "$WANT" = newest ]; then
  pick=$(tail -n 1 <<<"$versions")
elif [[ $WANT =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
  pick=$(grep -xF "$WANT" <<<"$versions" || true)
elif [[ $WANT =~ ^[0-9]+\.[0-9]+$ ]]; then
  pick=$(grep -E "^${WANT//./\\.}\.[0-9]+$" <<<"$versions" | tail -n 1 || true)
else
  echo "Version must be newest, major.minor or major.minor.build, not '$WANT'" >&2
  exit 1
fi
[ -n "$pick" ] || { echo "No release of $SOURCE/omnicore-api and omnicore-web matches '$WANT'" >&2; exit 1; }

# The version for the next steps: a file, and whatever the CI system passes between jobs.
echo "Deploying Archestack $pick"
mkdir -p "$OUT"
echo "$pick" > "$OUT/version"
if [ -n "${TF_BUILD:-}" ]; then echo "##vso[build.updatebuildnumber]$pick"; fi
if [ -n "${GITHUB_OUTPUT:-}" ]; then
  echo "version=$pick" >> "$GITHUB_OUTPUT"
  echo "Deploying Archestack **$pick**" >> "$GITHUB_STEP_SUMMARY"
fi
if [ -n "${GITLAB_CI:-}" ]; then echo "VERSION=$pick" > "$OUT/version.env"; fi

• Versions are compared as numbers, so 1.0.10 is newer than 1.0.9, and a missing number is skipped rather than waited for.
• Only a version both images carry counts. A release build that stopped between pushing the API and pushing the web image is never deployed by halves.
• An exact version is checked as well. A version the images do not have stops the run before anything is touched.
• The version travels the way each CI system passes values on: Azure DevOps names the run after it, GitHub Actions makes it a job output and shows it in the run's summary, and GitLab CI hands it to the later jobs as $VERSION.

deploy.sh then takes one environment to that version:

#!/usr/bin/env bash
# aks/deploy.sh: copy one Archestack version into your registry and roll it out with Helm.
# Every pipeline runs it once per environment, signed in to Azure. By hand, after az login:
#   VERSION=1.4.23 ENVIRONMENT=prd bash aks/deploy.sh
set -euo pipefail
cd "$(dirname "$0")/.."
: "${VERSION:?}" "${ENVIRONMENT:?}"
source aks/environment.sh
SOURCE=${SOURCE_NAMESPACE:-nickpeelman}
VALUES="aks/environments/$ENVIRONMENT/values.yaml"
if grep -v '^[[:space:]]*#' "$VALUES" | grep -q CHANGE-ME; then echo "Fill in every CHANGE-ME in $VALUES first" >&2; exit 1; fi

# Keeps a value out of the pipeline's log. GitLab has no such command; nothing here prints one.
mask() {
  if [ -n "${TF_BUILD:-}" ]; then echo "##vso[task.setsecret]$1"; fi
  if [ -n "${GITHUB_ACTIONS:-}" ]; then echo "::add-mask::$1"; fi
}

# 1. The two images, imported from Docker Hub into your registry. --force makes a rerun harmless.
for image in omnicore-api omnicore-web; do
  az acr import --name "$ACR" --force \
    --source "docker.io/$SOURCE/$image:$VERSION" --image "$image:$VERSION"
done

# 2. The passwords, from Key Vault into files only this job can read. They go in on every
#    upgrade: the chart writes the SMTP password from the values each time.
secret() { az keyvault secret show --vault-name "$KV" -n "$1" --query value -o tsv 2>/dev/null || true; }
pg=$(secret archestack-postgres-password)
smtp=$(secret archestack-smtp-password)
[ -n "$pg" ] || { echo "Cannot read archestack-postgres-password from $KV (missing, or no Key Vault Secrets User)" >&2; exit 1; }
for value in "$pg" "$smtp"; do
  if [ -n "$value" ]; then mask "$value"; fi
done
umask 077
secrets=$(mktemp -d)
trap 'rm -rf "$secrets"' EXIT
printf '%s' "$pg" > "$secrets/postgres"
printf '%s' "$smtp" > "$secrets/smtp"

# 3. The rollout, into the namespace setup/3-pipeline.sh made.
kube_login
helm upgrade --install archestack infra/helm/archestack \
  --namespace "$NAMESPACE" \
  -f "$VALUES" \
  --set-file postgres.password="$secrets/postgres" \
  --set-file smtp.password="$secrets/smtp" \
  --set-string postgres.host="$PG.postgres.database.azure.com" \
  --set-string 'caddy.service.annotations.service\.beta\.kubernetes\.io/azure-pip-name'="$PIP" \
  --set-string api.image.repository="$ACR.azurecr.io/omnicore-api" \
  --set-string api.image.tag="$VERSION" \
  --set-string web.image.repository="$ACR.azurecr.io/omnicore-web" \
  --set-string web.image.tag="$VERSION" \
  --wait --timeout 30m

• The images go into your registry first. Nodes pull from Azure, inside the region, so adding or replacing a node never waits on Docker Hub or its limits on anonymous pulls. Should an import be refused for those limits, give az acr import a Docker Hub account with --username and --password.
• The passwords go in on every upgrade. The chart writes the SMTP password from the values each time, so an upgrade without it leaves the instance unable to send mail. They travel in files only the job can read, and Azure DevOps and GitHub Actions mask them in the log.
• --wait --timeout 30m waits until every Deployment is ready. The API migrates every workspace before it answers, so a version with migrations takes a while on an instance with many workspaces.
• No --atomic. A failed rollout is left for a look rather than undone, because the new version may already have migrated the databases. The old pods keep serving meanwhile.

The pipeline

The kit has the same pipeline for three CI systems. Each runs the two scripts above, in the same order:

Archestack publishes 1.4.23     omnicore-api and omnicore-web, tagged 1.4.23, <commit> and latest

03:00, the nightly run
  resolve    newest -> 1.4.23   the highest number both images carry
  dev        import 1.4.23 into your registry, helm upgrade, wait until ready
  tst        the same 1.4.23
             a scheduled run stops here

a run started by hand, version 1.4.23
  dev, tst   already on 1.4.23, nothing rolls out
  acc        waits for its approval, then 1.4.23
  prd        waits for its approval, then 1.4.23

• Every night the schedule takes the newest release to development and test.
• A scheduled run stops after test. Acceptance and production move only on a run started by hand. Give it the version test runs, such as 1.4.23, and acceptance and production get exactly what was tested. Left at newest, it takes whatever is newest at that moment.
• One version per run. The version is resolved once, before the first environment, so a release published while a run waits for an approval does not slip into production.
• An environment already on the version rolls nothing out. The pods' specification is unchanged, so Kubernetes leaves them running. A change to its values.yaml is still applied.

# aks/azure-pipelines.yml: the newest Archestack release to dev and tst every night,
# to acc and prd on a run started by hand, once each is approved.
trigger: none

schedules:
  - cron: "0 3 * * *"              # 03:00 UTC
    displayName: Nightly, newest release to dev and tst
    branches:
      include: [main]
    always: true                    # a release is a new image, not a commit in this repository

parameters:
  - name: version
    displayName: Archestack version (newest, 1.4 for the newest 1.4.x, or 1.4.23)
    type: string
    default: newest
  - name: environments
    displayName: Environments, in order
    type: object
    default: [dev, tst, acc, prd]

pool:
  vmImage: ubuntu-latest

stages:
  # Once per run, so every environment gets the same number even when a release
  # is published while the run waits for an approval.
  - stage: resolve
    displayName: Resolve the version
    jobs:
      - job: resolve
        steps:
          - script: bash aks/resolve-version.sh
            displayName: Newest version both images carry
            env:
              ARCHESTACK_VERSION: ${{ parameters.version }}
              OUT_DIR: $(Build.ArtifactStagingDirectory)
          - publish: $(Build.ArtifactStagingDirectory)
            artifact: version

  - ${{ each env in parameters.environments }}:
    - stage: deploy_${{ env }}
      displayName: Deploy ${{ env }}
      ${{ if in(env, 'acc', 'prd') }}:
        # A scheduled run stops after tst.
        condition: and(succeeded(), ne(variables['Build.Reason'], 'Schedule'))
      jobs:
        - deployment: deploy
          environment: archestack-${{ env }}
          strategy:
            runOnce:
              deploy:
                steps:
                  - checkout: self
                  - download: current
                    artifact: version
                  - task: AzureCLI@2
                    displayName: Import the images, then helm upgrade
                    inputs:
                      azureSubscription: sc-archestack-${{ env }}
                      scriptType: bash
                      scriptLocation: inlineScript
                      workingDirectory: $(Build.SourcesDirectory)
                      inlineScript: |
                        export VERSION=$(cat "$PIPELINE_WORKSPACE/version/version")
                        ENVIRONMENT=${{ env }} bash aks/deploy.sh

# .github/workflows/archestack.yml: the newest Archestack release to dev and tst every night,
# to acc and prd on a run started by hand, once each is approved.
name: Archestack on AKS

on:
  schedule:
    - cron: "0 3 * * *"            # 03:00 UTC
  workflow_dispatch:
    inputs:
      version:
        description: Archestack version (newest, 1.4 for the newest 1.4.x, or 1.4.23)
        default: newest

permissions:
  id-token: write                  # sign in to Azure with a token, no stored secret
  contents: read

jobs:
  # Once per run, so every environment gets the same number even when a release
  # is published while the run waits for an approval.
  resolve:
    runs-on: ubuntu-latest
    outputs:
      version: ${{ steps.pick.outputs.version }}
    steps:
      - uses: actions/checkout@v4
      - id: pick
        run: bash aks/resolve-version.sh
        env:
          ARCHESTACK_VERSION: ${{ inputs.version || 'newest' }}
          OUT_DIR: ${{ runner.temp }}

  dev:
    needs: resolve
    uses: ./.github/workflows/archestack-deploy.yml
    with: { environment: dev, version: "${{ needs.resolve.outputs.version }}" }

  tst:
    needs: [resolve, dev]
    uses: ./.github/workflows/archestack-deploy.yml
    with: { environment: tst, version: "${{ needs.resolve.outputs.version }}" }

  # A scheduled run stops after tst.
  acc:
    if: github.event_name != 'schedule'
    needs: [resolve, tst]
    uses: ./.github/workflows/archestack-deploy.yml
    with: { environment: acc, version: "${{ needs.resolve.outputs.version }}" }

  prd:
    if: github.event_name != 'schedule'
    needs: [resolve, acc]
    uses: ./.github/workflows/archestack-deploy.yml
    with: { environment: prd, version: "${{ needs.resolve.outputs.version }}" }

# .github/workflows/archestack-deploy.yml: one environment to one version, called by archestack.yml.
name: Archestack deploy

on:
  workflow_call:
    inputs:
      environment: { type: string, required: true }
      version: { type: string, required: true }

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: archestack-${{ inputs.environment }}
    concurrency: archestack-${{ inputs.environment }}    # one rollout per environment at a time
    steps:
      - uses: actions/checkout@v4
      - uses: azure/login@v2
        with:
          client-id: ${{ vars.AZURE_CLIENT_ID }}       # this environment's identity
          tenant-id: ${{ vars.AZURE_TENANT_ID }}
          subscription-id: ${{ vars.AZURE_SUBSCRIPTION_ID }}
      - run: bash aks/deploy.sh
        env:
          VERSION: ${{ inputs.version }}
          ENVIRONMENT: ${{ inputs.environment }}

# .gitlab-ci.yml: the newest Archestack release to dev and tst on the nightly schedule,
# to acc and prd from a pipeline started by hand, each when a person runs its job.
variables:
  ARCHESTACK_VERSION:
    value: newest
    description: Archestack version (newest, 1.4 for the newest 1.4.x, or 1.4.23)

# Pipelines run on the schedule (Build, Pipeline schedules) and when started by hand, never on a push.
workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule" || $CI_PIPELINE_SOURCE == "web"

stages: [resolve, dev, tst, acc, prd]

default:
  image: mcr.microsoft.com/azure-cli:2.90.0
  before_script:
    - tdnf install -y curl tar gzip > /dev/null

# Once per pipeline, so every environment gets the same number. Later jobs read it as $VERSION.
resolve:
  stage: resolve
  script:
    - bash aks/resolve-version.sh
  artifacts:
    reports:
      dotenv: version.env

.deploy:
  id_tokens:
    AZURE_TOKEN:
      aud: https://gitlab.com          # GITLAB_URL from aks/settings.env, on a self-managed instance
  before_script:
    - tdnf install -y curl tar gzip > /dev/null
    - curl -fsSL https://get.helm.sh/helm-v3.16.2-linux-amd64.tar.gz | tar -xz -C /tmp
    - mv /tmp/linux-amd64/helm /usr/local/bin/helm
    - az login --service-principal -u "$AZURE_CLIENT_ID" -t "$AZURE_TENANT_ID" --federated-token "$AZURE_TOKEN" -o none
    - az account set -s "$AZURE_SUBSCRIPTION_ID"
  script:
    - bash aks/deploy.sh

dev:
  extends: .deploy
  stage: dev
  environment: archestack-dev
  resource_group: archestack-dev      # one rollout per environment at a time
  variables: { ENVIRONMENT: dev }

tst:
  extends: .deploy
  stage: tst
  environment: archestack-tst
  resource_group: archestack-tst
  variables: { ENVIRONMENT: tst }

# A scheduled pipeline stops after tst. acc and prd wait until a person runs them, in order.
.by-hand:
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"
      when: never
    - when: manual
  allow_failure: false                # prd stays blocked until acc has run

acc:
  extends: [.deploy, .by-hand]
  stage: acc
  environment: archestack-acc
  resource_group: archestack-acc
  variables: { ENVIRONMENT: acc }

prd:
  extends: [.deploy, .by-hand]
  stage: prd
  environment: archestack-prd
  resource_group: archestack-prd
  variables: { ENVIRONMENT: prd }

Upgrades without downtime

• One new API pod at a time. The chart adds a pod before it removes one (maxSurge: 1, maxUnavailable: 0), and an old pod leaves only once a new one is ready.
• Start-up takes turns. Replicas that start together prepare the databases one after another under a lock, so two of them never migrate at once. A startup probe gives each pod 15 minutes before Kubernetes gives up on it.
• Node upgrades keep the API up. A PodDisruptionBudget keeps at least one API pod running while AKS drains a node.
• Caddy restarts only when its own configuration changes, such as a new hostname. It is one pod with its certificates on a disk of its own, so that restart is a short gap. An Archestack upgrade does not touch it.

Going back

Run the pipeline with the version before, such as 1.4.22, or roll the release back with Helm:

helm history archestack -n archestack                          # the revisions, newest last
helm rollback archestack -n archestack --wait --timeout 30m     # back to the one before

Either way, the databases stay as the newer version left them: it migrated them when it started, and an older version does not undo that. Going back runs older code on a newer schema, so prefer the next release that fixes the problem. Taking the schema back as well means restoring the server with Flexible Server's point-in-time restore.

Gotchas

• The chart comes from your repository, the images from the registry. The pipeline takes the newest images by itself, never a newer chart. When you move to a new major or minor version, take the chart from the Archestack repository again.
• The observability images carry no version number. They are tagged with a commit and latest, come from Docker Hub and follow observability.imageTag. Set it to one of their commit tags to hold them still.
• One release per namespace. Grafana and Tempo find Prometheus, Tempo and Loki by fixed service names.
• Behind PgBouncer (Flexible Server's port 6432), set postgres.clusterConnectionString to a direct connection on 5432. The replicas talk to each other over a PostgreSQL connection that has to stay open.
• Volumes and the chart's own Secret survive helm uninstall. Delete them by hand when you mean it.

Where this goes next

Capacity and sizing has the measured figures for sizing the cluster and the database. Workspaces and hosting explains how one instance serves many workspaces. Once the instance runs, Deployment strategy covers deploying your app into its workspaces from a repository.