Fragmented Thought

Creating subscribable Azure DevOps versioned pipeline artifacts

By

Published

Lance Gliser

Creating pipelines is a funny business. There are several concerns to watch for that would be standard in version control, but require special work instead. I'd like to review the idea of a SemVer'esk major version control system today. I'll be covering Azure DevOps pipelines, specifically, multi-stage deployment artifact pipelines with templates. If you need a general primer, I highly recommend their Create a multistage pipeline by using Azure Pipelines module. It will cover the basics of yaml, stages, jobs, tasks, and deployments that I'll only glance over here.

The example module is targeted at one repository, one pipeline, and a few deployment targets all known by the pipeline authors. It breaks down once you start producing generic artifacts without knowing what teams or projects might consume them. A few use cases we've found so far all relate to wrapping our own standards around generic open source projects. Thus far, we've wrapped ArangoDB, Prometheus monitoring stack, Elastic stack, and other helm charts with variable sets.

Setting up your publishing pipeline

Generic artifacts created by pipelines do not inherit git tags. They do however allow you to manually tag the artifact runs using the "more actions triple dot"

The tags work a little differently than standard git usages:

  • You can use the same tag multiple times. We'll be using that feature for publishing!
  • Any subscribers will pick up the latest pipeline with the specified tag. If you tag an old release later, you can cause regressions.
  • While you can make full SemVer tags with 3 octets, subscribers can not use them. For this reason, we'll use major tags such 1.x, 2.x, etc to manage subscribers.

Each time your major tag is added to a release, subscribers will have the option of automatically trigger a new run.

Here's an example publishing pipeline. I tend to use azure-pipelines/release.yaml:

# Add steps that build, run tests, deploy, and more: # https://aka.ms/yaml trigger: batch: true branches: include: - main variables: vmImage: ubuntu-latest stages: - stage: Publish displayName: "Publish the base artifact" jobs: - job: "Publish" pool: vmImage: $(vmImage) displayName: "Prepare job" steps: - task: CopyFiles@2 displayName: "Copy helm charts" inputs: Contents: "helm" TargetFolder: $(Build.ArtifactStagingDirectory) - task: CopyFiles@2 displayName: "Copy helm charts" inputs: Contents: | helm/** README.md TargetFolder: $(Build.ArtifactStagingDirectory) - task: PublishBuildArtifacts@1 inputs: PathtoPublish: $(Build.ArtifactStagingDirectory) ArtifactName: "drop" publishLocation: "Container"

Setting up your subscription pipelines

Your pipeline produces an artifact that can be subscribed to from other project's pipelines and customized for specific use. As an example, I'll include parts of our ArangoDB helm charts.

To do so:

  • Create your own new repository in the consuming project.
  • Create a subscriber pipeline. I tend to use azure-pipelines/deployment/pipeline.yml
  • Include a build resource dependency on your publishing build artifact:
resources: pipelines: - pipeline: _ArangoChart # Used to download the artifact to local files: $(Pipeline.Workspace)/_ArangoChart project: Deployment and Hosting # Any project name source: arango-chart # The actual pipeline in the project # One or more tags. # You could use "latest", or any static tag. # I would suggest major versions to control breaking changes. tags: - 2.x # Include a trigger to allow automatic updating if desired trigger: tags: - 2.x
  • Include a deployment stage
variables: vmImage: ubuntu-latest image: arangodb/arangodb:3.7.6 releaseName: some-release-name stages: - stage: Development displayName: "Deploy to the dev environment" jobs: - deployment: Deploy pool: vmImage: $(vmImage) environment: some-cluster.some-resource variables: - name: hostName value: "dev.arango.example.com" readonly: true strategy: runOnce: deploy: steps: - template: "templates/helm-db-upgrade.yml" parameters: namespace: $(Environment.ResourceName) releaseName: $(releaseName) environment: Development image: $(image) hostname: $(hostName) agentsCount: 2 dbserversCount: 2 coordinatorsCount: 2 - stage: Production displayName: "Deploy to the production environment" jobs: - deployment: Deploy pool: vmImage: $(vmImage) environment: some-cluster.some-resource variables: - name: hostName value: "arango.example.com" readonly: true strategy: runOnce: deploy: steps: - template: "templates/helm-db-upgrade.yml" parameters: namespace: $(Environment.ResourceName) releaseName: $(releaseName) image: $(image) hostname: $(hostName)
  • Include a step in your azure-pipelines/deployment/templates/helm-db-upgrade.yml to install this chart and customize the values as required.
parameters: - name: namespace type: string - name: releaseName type: string - name: mode type: string default: Cluster values: - Cluster - ActiveFailover - Single - name: environment type: string default: Production values: - Development - Production - name: image type: string - name: imagePullPolicy type: string default: Always values: - Always - IfNotPresent - name: hostName type: string - name: agentsCount type: number # Suggested development value: 1 default: 3 # Based on mode: Cluster defaults - name: dbserversCount type: number # Suggested development value: 2 default: 3 # Based on mode: Cluster defaults - name: coordinatorsCount type: number # Suggested development value: 1 default: 3 # Based on mode: Cluster defaults steps: # Ensure the operator is deployed - task: HelmDeploy@0 displayName: "helm upgrade some-arango-operator" inputs: namespace: ${{ parameters.namespace }} # The K8s namespace command: upgrade chartType: FilePath chartPath: "$(Pipeline.Workspace)/_ArangoChart/drop/helm/operator" releaseName: ${{ parameters.releaseName }}-db-operator waitForExecution: true # Ensure the instance is deployed - task: HelmDeploy@0 displayName: "helm upgrade some-arango-instance" inputs: namespace: ${{ parameters.namespace }} # The K8s namespace command: upgrade chartType: FilePath chartPath: "$(Pipeline.Workspace)/_ArangoChart/drop/helm/instance" releaseName: ${{ parameters.releaseName }}-db-instance overrideValues: "spec.mode=${{ parameters.mode }},\ spec.environment=${{ parameters.environment }},\ spec.image=${{ parameters.image }},\ spec.imagePullPolicy=${{ parameters.imagePullPolicy }},\ spec.agents.count=${{ parameters.agentsCount }},\ spec.dbserversCount.count=${{ parameters.dbserversCount }},\ spec.coordinatorsCount.count=${{ parameters.coordinatorsCount }},\ ingress.hosts[0].host=${{ parameters.hostName }},\ ingress.tls[0].hosts[0]=${{ parameters.hostName }},\ ingress.tls[0].secretName=${{ parameters.hostName }}-tls" waitForExecution: false