Guide · 12 min

Deploying from the repository

The repository is the source of an Archestack app, the way source code is the source of a program. A deployment treats Archestack as the compiler: a pipeline builds a throwaway Archestack from the repository's files, runs the tests against it, and only when they pass does it plan the change against the hosted environment and apply it there. The hosted environment keeps serving its users throughout.

Status: the Archestack CLI runs every stage below, from a script step in any pipeline. Ready-made pipeline tasks for Azure DevOps and GitHub Actions that wrap it are in development; What runs today says what each stage runs.

Your CI system

The pipeline

merge to main
  build     a throwaway Archestack made from the repository's files
  test      every package's test cases, each in a sandbox
  plan      against tst: artifacts and SQL, with a plan hash
  deploy    tst: schema, then files, then the release
  approve   the plan against acc, then deploy acc
  approve   the plan against prd, then deploy prd

A pull request runs the build, the tests and the plan against test, so a reviewer sees what would change before anything is merged. A merge to the main branch runs every stage, and acceptance and production each wait for a person to approve their plan.

1. Build: Archestack as the compiler

The pipeline starts an empty Archestack on the build agent, in Docker from the same API image the environments run, or from a runtime folder like the one the VS Code extension runs projects with, and loads the repository into it: every package's tables in one deployment, then each package's artifacts, dependencies before the packages that need them. This is where a mistake shows up first, the way a compiler error does: a reference to a business entity, table or script that does not exist, or a name another package already uses, stops the load. The result is an environment made from the files alone, with nothing left over from anyone's machine.

2. Test

Every package's test cases run against the build, each in a sandbox database of its own, with mail and notifications recorded rather than sent. Every script a test runs is compiled first, so a script that no longer compiles fails its test. One failed case stops the pipeline, and nothing is deployed.

3. Plan against the target

The same files are planned against the target environment: which artifacts are created, changed or removed there, and which table changes run as SQL, with every operation's risk and the typed scripts it would stop compiling. The plan carries a hash. The deployment applies exactly the plan that was approved: if the target changed in the meantime, it is refused and the plan is made again.

4. Deploy: schema, files, release

  1. Schema. The table changes run as a database deployment, listed in the environment's deployment history like any other. A change that loses data runs only when the pipeline explicitly allows data loss.
  2. Files. The artifacts are written in one transaction, so users see all of a package's new version or none of it. A deployment touches only what the package holds and creates, and never what each environment decides for itself.
  3. Release. Once the files are in, the pipeline publishes a release by itself. That is the moment the new pages and app layouts reach the published app, and nobody publishes by hand. Every deployment is a new version of the app with a release of its own.

An environment that has never published a release shows its drafts, so the new files would be visible the moment they land. Before its first deployment to such an environment, the pipeline therefore publishes a release of what the environment already holds, and from then on the new files wait as drafts until their own release, as in every later deployment.

Why the environment stays up

  • Nothing restarts. A deployment is data in the environment's databases, not a new build of the server. The API keeps answering throughout, and a changed script is compiled the first time it runs.
  • Files land at once. One transaction per package, so nobody works against half a package.
  • Pages switch in one step. The published app reads the current release. New pages wait as drafts until the release is published, which moves every user to them at the same moment. Activating the previous release rolls back; that restores the pages and layouts, never data or tables.
  • Table changes are planned and reviewed. Adding a table or a nullable column does not get in anyone's way, and anything that loses data needs an explicit yes.

Expand, then contract

Script modules, business events and business entities are live the moment their files land, not at the release. A change that renames or removes a column therefore goes out in two deployments:

  1. Expand. Add the new table or column, and deploy the scripts and pages that use it, with the old one still in place.
  2. Contract. In a later deployment, once nothing reads the old column any more, drop it. The plan marks the drop as data loss, and the typed script check says whether anything still uses it.

A change that rewrites a large table, such as a new type for a column, holds a lock on that table while it runs. Schedule those for a quiet moment.

Environments

The repository lists its environments in archestack.json. An environment marked "mode": "pipeline" is deployed only by the pipeline, and the VS Code extension refuses to push to it by hand: 

"environments": {
  "tst": { "url": "https://acme-tst.example.com", "apiUrl": "https://api.example.com", "workspace": "acme-tst.example.com", "mode": "pipeline" },
  "acc": { "url": "https://acme-acc.example.com", "apiUrl": "https://api.example.com", "workspace": "acme-acc.example.com", "mode": "pipeline" },
  "prd": { "url": "https://acme.example.com", "apiUrl": "https://api.example.com", "workspace": "acme.example.com", "mode": "pipeline" }
}

What each environment decides for itself is not in the files and not in any deployment: users and roles, page permissions, public flags, Data API grants and keys, mail accounts and connections to other systems. Each environment's administrators set them, so a deployment can never make something public or give anyone access.

The pipeline file

The pipeline lives in the repository and deploys from it: every run checks out one commit, and those files are what is built, tested and deployed. Nothing is exported from a workspace. Below is the same pipeline for three CI systems, each taking a change to test, acceptance and production. The archestack commands are those of the Archestack CLI, so any other CI system that runs .NET and Docker on Linux runs the same steps.

  • Build and test runs for every pull request and every merge to main. It builds a throwaway Archestack from the commit, runs every package's tests, and packs the repository into one artifact, so every environment gets exactly the same files.
  • A plan and a deploy step per environment. The plan is made against the environment and shown in the run; the deploy then applies exactly that plan, after an approval for acceptance and production. Environments go one after another, so acceptance is planned only once test is deployed, and production only once acceptance is. A pull request stops after the plan against test.
  • Deploy tokens. Each environment's token is made in that environment's builder, under Operations, Deploy Tokens. It is shown once, acts as an admin, and works only for deploying: it never reads the environment's data. The environment's address comes from archestack.json.
Your CI system

Azure DevOps

The pipeline is azure-pipelines.yml at the repository's root:

# azure-pipelines.yml: build and test, then plan and deploy test, acceptance and production
trigger:
  branches:
    include: [main]

pr:
  branches:
    include: [main]

parameters:
  - name: environments
    type: object
    default: [tst, acc, prd]
  - name: allowDataLoss
    displayName: Allow changes that lose data
    type: boolean
    default: false

pool:
  vmImage: ubuntu-latest

stages:
  - stage: build
    displayName: Build and test
    jobs:
      - job: build
        steps:
          - checkout: self
            displayName: The repository at this commit
          - script: |
              dotnet tool install --global Archestack.Cli
              echo "##vso[task.prependpath]$HOME/.dotnet/tools"
            displayName: Install the Archestack CLI
          - script: archestack build
            displayName: Build a throwaway Archestack from the repository
          - script: archestack test --junit $(Agent.TempDirectory)/tests.xml
            displayName: Run every package's tests
          - task: PublishTestResults@2
            condition: succeededOrFailed()
            inputs:
              testResultsFormat: JUnit
              testResultsFiles: $(Agent.TempDirectory)/tests.xml
          - script: archestack pack --out $(Build.ArtifactStagingDirectory)/app.zip
            displayName: Pack the repository
          - publish: $(Build.ArtifactStagingDirectory)
            artifact: app

  - ${{ each env in parameters.environments }}:
    - stage: plan_${{ env }}
      displayName: Plan ${{ env }}
      ${{ if ne(env, 'tst') }}:
        condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))
      variables:
        - group: archestack-${{ env }}
      jobs:
        - job: plan
          steps:
            - download: current
              artifact: app
            - script: |
                dotnet tool install --global Archestack.Cli
                echo "##vso[task.prependpath]$HOME/.dotnet/tools"
              displayName: Install the Archestack CLI
            - script: >-
                archestack plan --env ${{ env }}
                --from $(Pipeline.Workspace)/app/app.zip
                --allow-data-loss ${{ parameters.allowDataLoss }}
                --out $(Build.ArtifactStagingDirectory)/plan.json
                --summary $(Build.ArtifactStagingDirectory)/plan.md
              displayName: Plan against ${{ env }}
              env:
                ARCHESTACK_TOKEN: $(ARCHESTACK_TOKEN)
            - script: echo "##vso[task.uploadsummary]$(Build.ArtifactStagingDirectory)/plan.md"
              displayName: Show the plan in the run
            - publish: $(Build.ArtifactStagingDirectory)
              artifact: plan-${{ env }}

    - stage: deploy_${{ env }}
      displayName: Deploy ${{ env }}
      condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))
      variables:
        - group: archestack-${{ env }}
      jobs:
        - deployment: deploy
          environment: archestack-${{ env }}
          strategy:
            runOnce:
              deploy:
                steps:
                  - download: current
                    artifact: app
                  - download: current
                    artifact: plan-${{ env }}
                  - script: |
                      dotnet tool install --global Archestack.Cli
                      echo "##vso[task.prependpath]$HOME/.dotnet/tools"
                    displayName: Install the Archestack CLI
                  - script: >-
                      archestack deploy --env ${{ env }}
                      --from $(Pipeline.Workspace)/app/app.zip
                      --plan $(Pipeline.Workspace)/plan-${{ env }}/plan.json
                    displayName: Schema, files and release
                    env:
                      ARCHESTACK_TOKEN: $(ARCHESTACK_TOKEN)
  • Approvals. Create the environments archestack-tst, archestack-acc and archestack-prd under Pipelines, Environments, and add an Approvals check to acceptance and production. Azure DevOps then waits for an approver before those deploy stages start, with the plan in the run's summary. The test results are published to the run.
  • Credentials. Each environment's deploy token is the secret ARCHESTACK_TOKEN in a variable group of the same name as the environment, such as archestack-prd.
  • Data loss. Allow changes that lose data is a checkbox when a run is started by hand, off by default. Without it, a plan that drops a table or a column stops the pipeline.
  • Pull requests on Azure Repos. Azure Repos ignores the pr: section; add the pipeline as a build validation branch policy on main instead.
  • A pipeline made in the web editor is saved as a new file named after the pipeline. Point the pipeline at azure-pipelines.yml under its settings, or the file you edit is not the one that runs.

GitHub Actions

The workflow is .github/workflows/archestack.yml:

# .github/workflows/archestack.yml: build and test, then plan and deploy test, acceptance and production
name: Archestack

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
  workflow_dispatch:
    inputs:
      allow-data-loss:
        description: Allow changes that lose data
        type: boolean
        default: false

permissions:
  contents: read

jobs:
  build:
    name: Build and test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-dotnet@v4
        with:
          dotnet-version: 8.0.x
      - name: Install the Archestack CLI
        run: dotnet tool install --global Archestack.Cli
      - name: Build a throwaway Archestack from the repository
        run: archestack build
      - name: Run every package's tests
        run: archestack test --junit "$RUNNER_TEMP/tests.xml"
      - uses: actions/upload-artifact@v4
        if: ${{ !cancelled() }}
        with:
          name: tests
          path: ${{ runner.temp }}/tests.xml
      - name: Pack the repository
        run: archestack pack --out "$RUNNER_TEMP/app/app.zip"
      - uses: actions/upload-artifact@v4
        with:
          name: app
          path: ${{ runner.temp }}/app

  # A pull request stops after the plan against test.
  tst:
    needs: build
    uses: ./.github/workflows/archestack-deploy.yml
    with:
      environment: tst
      deploy: ${{ github.event_name != 'pull_request' }}
      allow-data-loss: ${{ inputs.allow-data-loss == true }}
    secrets:
      token: ${{ secrets.ARCHESTACK_TOKEN_TST }}

  acc:
    if: github.event_name != 'pull_request'
    needs: tst
    uses: ./.github/workflows/archestack-deploy.yml
    with:
      environment: acc
      allow-data-loss: ${{ inputs.allow-data-loss == true }}
    secrets:
      token: ${{ secrets.ARCHESTACK_TOKEN_ACC }}

  prd:
    if: github.event_name != 'pull_request'
    needs: acc
    uses: ./.github/workflows/archestack-deploy.yml
    with:
      environment: prd
      allow-data-loss: ${{ inputs.allow-data-loss == true }}
    secrets:
      token: ${{ secrets.ARCHESTACK_TOKEN_PRD }}

Each environment is planned and deployed by the same reusable workflow beside it:

# .github/workflows/archestack-deploy.yml: plans the packed files against one environment, then deploys that plan
name: Archestack environment

on:
  workflow_call:
    inputs:
      environment:
        type: string
        required: true
      deploy:
        type: boolean
        default: true
      allow-data-loss:
        type: boolean
        default: false
    secrets:
      token:
        required: true

jobs:
  plan:
    name: Plan ${{ inputs.environment }}
    runs-on: ubuntu-latest
    steps:
      - uses: actions/download-artifact@v4
        with:
          name: app
          path: app
      - uses: actions/setup-dotnet@v4
        with:
          dotnet-version: 8.0.x
      - run: dotnet tool install --global Archestack.Cli
      - name: Plan against ${{ inputs.environment }}
        run: >-
          archestack plan --env ${{ inputs.environment }}
          --from app/app.zip
          --allow-data-loss ${{ inputs.allow-data-loss }}
          --out plan/plan.json
          --summary plan/plan.md
        env:
          ARCHESTACK_TOKEN: ${{ secrets.token }}
      - name: Show the plan in the run
        if: ${{ !cancelled() }}
        run: cat plan/plan.md >> "$GITHUB_STEP_SUMMARY"
      - uses: actions/upload-artifact@v4
        with:
          name: plan-${{ inputs.environment }}
          path: plan

  deploy:
    name: Deploy ${{ inputs.environment }}
    if: inputs.deploy
    needs: plan
    runs-on: ubuntu-latest
    environment: archestack-${{ inputs.environment }}
    concurrency: archestack-${{ inputs.environment }}
    steps:
      - uses: actions/download-artifact@v4
        with:
          name: app
          path: app
      - uses: actions/download-artifact@v4
        with:
          name: plan-${{ inputs.environment }}
          path: plan
      - uses: actions/setup-dotnet@v4
        with:
          dotnet-version: 8.0.x
      - run: dotnet tool install --global Archestack.Cli
      - name: Schema, files and release
        run: archestack deploy --env ${{ inputs.environment }} --from app/app.zip --plan plan/plan.json
        env:
          ARCHESTACK_TOKEN: ${{ secrets.token }}
  • Approvals. Create the environments archestack-tst, archestack-acc and archestack-prd under Settings, Environments, and add required reviewers to acceptance and production. The deploy job waits for a reviewer, with the plan already in the run's summary. On a private repository, environments need GitHub Pro, Team or Enterprise.
  • Credentials. The deploy tokens are the repository secrets ARCHESTACK_TOKEN_TST, ARCHESTACK_TOKEN_ACC and ARCHESTACK_TOKEN_PRD, so the plan can run before the approval. A workflow on any branch of the repository can read a repository secret; to keep production's token to approved runs, make it an environment secret of archestack-prd instead, and its plan then runs after the approval.
  • Data loss. Allow changes that lose data is a checkbox under Run workflow, off by default.
  • Pull requests from forks get no secrets, so their plan against test fails; pull requests from branches of the repository itself plan as usual.
  • concurrency keeps two runs from deploying to one environment at the same time. The test results are kept as the run's tests artifact.

GitLab CI

The pipeline is .gitlab-ci.yml at the repository's root:

# .gitlab-ci.yml: build and test, then plan and deploy test, acceptance and production
workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

variables:
  ALLOW_DATA_LOSS:
    value: "false"
    description: Allow changes that lose data (true or false)

stages: [build, tst, acc, prd]

default:
  image: mcr.microsoft.com/dotnet/sdk:8.0
  before_script:
    - dotnet tool install --global Archestack.Cli
    - export PATH="$PATH:$HOME/.dotnet/tools"

build:
  stage: build
  # The throwaway Archestack runs in Docker, here a Docker service beside the job.
  services:
    - docker:27-dind
  variables:
    DOCKER_HOST: tcp://docker:2375
    DOCKER_TLS_CERTDIR: ""
  script:
    - curl -fsSL https://download.docker.com/linux/static/stable/x86_64/docker-27.3.1.tgz | tar -xz -C /usr/local/bin --strip-components=1 docker/docker
    - archestack build
    - archestack test --junit tests.xml
    - archestack pack --out app.zip
  artifacts:
    when: always
    paths: [app.zip]
    reports:
      junit: tests.xml

.plan:
  script:
    - archestack plan --env "$TARGET" --from app.zip --allow-data-loss "$ALLOW_DATA_LOSS" --out plan.json --summary plan.md
  artifacts:
    when: always
    expose_as: plan
    paths: [plan.md, plan.json]

.deploy:
  script:
    - archestack deploy --env "$TARGET" --from app.zip --plan plan.json

# A merge request stops after the plan against test.
plan-tst:
  extends: .plan
  stage: tst
  needs: [build]
  variables: { TARGET: tst }

deploy-tst:
  extends: .deploy
  stage: tst
  needs: [build, plan-tst]
  environment: archestack-tst
  resource_group: archestack-tst
  variables: { TARGET: tst }
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

plan-acc:
  extends: .plan
  stage: acc
  needs: [build, deploy-tst]
  variables: { TARGET: acc }
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

deploy-acc:
  extends: .deploy
  stage: acc
  needs: [build, plan-acc]
  environment: archestack-acc
  resource_group: archestack-acc
  variables: { TARGET: acc }
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: manual
  allow_failure: false

plan-prd:
  extends: .plan
  stage: prd
  needs: [build, deploy-acc]
  variables: { TARGET: prd }
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

deploy-prd:
  extends: .deploy
  stage: prd
  needs: [build, plan-prd]
  environment: archestack-prd
  resource_group: archestack-prd
  variables: { TARGET: prd }
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: manual
  allow_failure: false
  • Docker. The build runs the throwaway Archestack on a docker:dind service beside the job, which needs a runner that allows privileged containers, as GitLab.com's own runners do. DOCKER_HOST tells the CLI where Docker is, and it reaches the Archestack there.
  • Approvals. Deploying acceptance and production are manual jobs that block the pipeline, each started once its plan has been read. Every plan is kept with its job, and the plan against test is also linked from the merge request. On GitLab Premium, also protect those environments and require approvals.
  • Credentials. The deploy tokens are the masked CI/CD variables ARCHESTACK_TOKEN_TST, ARCHESTACK_TOKEN_ACC and ARCHESTACK_TOKEN_PRD. Protect acceptance's and production's, so only the main branch reads them. A protected variable does not reach a merge request pipeline, so leave test's unprotected for merge requests to plan against test.
  • Data loss. Run the pipeline by hand with ALLOW_DATA_LOSS set to true. Test results show in the merge request.
  • resource_group keeps two pipelines from deploying to one environment at the same time.

The Archestack CLI

The CLI is the .NET tool Archestack.Cli, installed with dotnet tool install --global Archestack.Cli on any agent with .NET 8 or later. Its commands run from the repository's root, or take --from with a folder or a packed ZIP:

  • archestack build starts the throwaway Archestack and loads the repository into it. It runs in Docker from the image the project section of archestack.json names, else the latest published one; --image picks another, and --runtime runs a runtime folder without Docker. It is ready once the NuGet packages the repository names are restored, and keeps running for the next step until archestack stop. --nuget-cache <folder> keeps those downloads for the next build, which otherwise downloads them again; two builds running at once each need a folder of their own.
  • archestack test runs every package's test cases against it, four at a time, and writes JUnit with --junit.
  • archestack pack --out app.zip packs archestack.json, packages/ and workspace/. The same files always give the same bytes, and the plan records their hash, so a deployment refuses files other than the ones planned.
  • archestack plan and archestack deploy work only on an environment marked "mode": "pipeline", with its deploy token in ARCHESTACK_TOKEN_<ENV> (such as ARCHESTACK_TOKEN_PRD) or ARCHESTACK_TOKEN. An artifact deleted from the repository is taken out of its package and left in the environment; plan with --delete-removed true to delete it there too. A package in the environment that the repository does not hold is left alone and named in the plan. When the repository names NuGet packages, both first ask the environment whether they restore there, and deploy waits up to five minutes after the last push until the environment has restored them, before it publishes the release. If that wait fails, the files are deployed but no release is published.

The exit code says what stopped a step: 3 the token was refused, 4 the environment changed since the plan (plan again), 5 the files cannot be applied as they stand, 6 the plan loses data and that was not allowed, 7 a test failed, 9 the server or the network failed, 10 the environment runs another major version of Archestack than the CLI, or one without NuGet support. NuGet packages that do not restore with the files are a 5, and so is an environment that has NuGet packages or typed data access turned off. For plan and deploy, a NuGet failure the files cannot fix, such as a server short of disk space, a feed it cannot reach, or a restore after the last push that failed or did not finish in five minutes, is a 9, so the same step can pass when it runs again.

What runs today

  • Build, test, plan and deploy run on the CLI, from script steps as in the pipelines above. On a developer's machine the VS Code extension's project is the same build: Reset Project Database makes it from nothing, and Run Tests runs every package's tests against it.
  • Pipeline tasks for Azure DevOps and GitHub Actions that wrap the CLI: in development.

Gotchas

  • A rollback does not undo tables or data. Every deployment is a release, so any version the pipeline deployed can be activated again, but that restores pages and layouts only. Taking tables back is a deployment of its own, planned and reviewed like any other.
  • Script modules and business events do not wait for the release. They are live the moment their files land, which is why a change that removes something goes out in two deployments.
  • Every environment downloads its own NuGet packages. The NuGet packages a package's nuget.json declares are downloaded by each environment that receives it, the build included, so each needs to reach nuget.org or the feed its servers are configured with. The build takes Scripting__NuGet__Sources and the Scripting__NuGet__Feeds__* settings from the environment the CLI runs in. A push whose NuGet packages cannot be restored is refused and changes nothing: the CLI's plan checks that before anything is deployed, and build and deploy wait until they are restored.

Where this goes next

Working as a team covers how developers share the repository that the pipeline deploys. Workspaces and hosting explains the environments it deploys to, and Database Deployments the schema step.