Fragmented Thought
TwitterGitHub

Creating Azure DevOps repository subscription pipelines

Azure DevOps - Project specific pipeline triggered by repository resource tag
By

Published:

Lance Gliser

Heads up! This content is more than six months old. Take some time to verify everything still works as expected.

Creating pipelines is a funny business. There are several concerns to watch for that would be standard in version control, but require special work instead. I'd like to review the idea of a SemVer'esk major version control system today. I'll be covering Azure DevOps pipelines, specifically, multi-stage > deployment and templates.

That's how I felt almost a year and a half ago. The links to documentation are still useful, but the method was hopelessly complex due requirements we've gotten past. In this post, I'd like to show how easy life can be instead. Today's example will create subscriber pipelines delivering project specific environmental variables to deliver a generic API shipped to client environments.

Creating the generic code base

Examples will be based on an updated version of my previous guide to deploy a Typescript GraphQL API via CDK. There will be some differences, and I'll repost the pipeline specifics here to ensure there's no confusion as our code base has evolved a bit.

We'll be working a project structure that has this organization:

  • azure-pipelines - Validation pipeline for testing and deployment templates
  • bin - CDK project standard
  • lambda - The src code for the lambda API
    • package.json
  • lib - CDK project standard
  • test - CDK project standard
  • package.json

Deployment templates

azure-pipelines/deployment/templates/npm-update.yml

This ensure we're working on the latest version of npm. Hosted machines have an older version by default. This bit us a couple times as the handling of peerDependencies changed since then.

steps:
  - task: CmdLine@2
    inputs:
      script: sudo npm install npm@latest --location=global
    displayName: npm install npm@latest --location=global

azure-pipelines/deployment/templates/npm-ci.yml

This installs all the required node modules, both for the root CDK project, and any APIs it deploys.

parameters:
  # The location of the chart downloaded to the stage from the artifact
  - name: workingDirectory
    type: string
    default: $(Pipeline.Workspace)/drop

steps:
  - task: Npm@1
    inputs:
      command: ci
      workingDir: ${{ parameters.workingDirectory }}
    displayName: npm ci - root

  - task: Npm@1
    inputs:
      command: ci
      workingDir: ${{ parameters.workingDirectory }}/lambda
    displayName: npm ci - lambda

azure-pipelines/deployment/templates/aws-deploy.yml

This performs the actual CDK deployment.

parameters:
  - name: awsCredentials
    type: string
  - name: regionName
    type: string
  - name: cdkAwsArguments
    type: string
    default: ""
  - name: workingDirectory
    type: string
    default: $(Pipeline.Workspace)/drop

steps:
  - script: |
      echo "Installing packages"
      sudo npm install --location=global aws-cdk ts-node
    displayName: "Installing node lambda for CDK"

  - task: AWSShellScript@1
    inputs:
      awsCredentials: ${{ parameters.awsCredentials }}
      disableAutoCwd: true
      inlineScript: cdk deploy --ci --require-approval never ${{ parameters.cdkAwsArguments }}
      regionName: ${{ parameters.regionName }}
      scriptType: inline
      workingDirectory: ${{ parameters.workingDirectory }}
    displayName: aws cdk deploy

Create a subscriber

Consider maintainability

If you're already thinking in publish and subscription patterns, it makes sense to step back from our immediate work and consider a reliable intermediary. Subscribers will need to maintain .yaml files. That's a painful process over time unless you get out ahead of it. Let's give them an easy method for doing so. In the same project you've created your API, create a new project and we'll encourage subscribers to fork it into their projects to get up to speed, and allow upstream sync to handle any changes you make as the API author.

Writing a repository subscriber pipeline

I wish this were more complicated or impressive, but am so happy that it is not. Unlike the mess with artifact subscriptions, repository subscriptions are dreamy:

azure-pipelines/{RepositoryName}/pipeline.yaml

# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml

# Do not automatically trigger if definition files update.
# That would cause all pipelines in this repo to trigger.
trigger: none

resources:
  repositories:
    - repository: {RepositoryName} # The name used to reference this repository in the checkout step
      type: git
      name: {ProjectName}/{RepositoryName}
      trigger:  # CI trigger for this repository, no CI trigger if skipped (only works for Azure Repos)
        tags:
          include:
            - v0.*
            - v1.*

pool:
  vmImage: ubuntu-latest
  demands:
    - npm

variables:
  - name: logLevel
    value: info
  - name: subdomain
    value: auth-api

stages:
  - stage: Dev
    displayName: Deploy to the Dev environment
    jobs:
      - deployment: Deploy
        environment: {EnvironmentName} # The string name of an Azure DevOps environment
        variables:
          # TODO define variable group for AWS variables in the consuming project
          - group: "" # The name of the variable group
          - name: awsHostedZoneId
            value: $[variables.AWS_RT53_HOSTED_ZONE_ID]
          - name: awsHostedZoneDomain
            value: $[variables.AWS_RT53_HOSTED_ZONE]
          - name: awsRegion
            value: $[variables.AWS_REGION]
          - name: awsAccessKey
            value: $[variables.AWS_ACCESS_KEY_ID]
          - name: awsSecretKey
            value: $[variables.AWS_SECRET_ACCESS_KEY]
        strategy:
          runOnce:
            deploy:
              steps:
                # Because this is a single repository checkout, $(Build.SourcesDirectory) is just this repo.
                - checkout: {RepositoryName}
                - template: azure-pipelines/deployment/templates/npm-update.yml@{RepositoryName}
                - template: azure-pipelines/deployment/templates/npm-ci.yml@{RepositoryName}
                  parameters:
                    workingDirectory: $(Build.SourcesDirectory)
                - template: azure-pipelines/deployment/templates/aws-deploy.yml@{RepositoryName}
                  parameters:
                    awsCredentials: "" # TODO update to service connection in project
                    cdkAwsArguments: >-
                      --context areAlarmsEnabled=$(areAlarmsEnabled)
                      --context hostedZoneId=$(awsHostedZoneId)
                      --context hostedZoneDomain=$(awsHostedZoneDomain)
                      --context logLevel=$(logLevel)
                      --context subdomain=$(subdomain)
                    regionName: $(awsRegion)
                    workingDirectory: $(Build.SourcesDirectory)

A special note about cdkAwsArguments: The >- yaml concats the lines into a single string for script args. You can't have comments, nor extra lines of white space in it.

Trigger considerations

  • Pipelines can not naturally reach into other projects. You must change a setting:
    Project > Settings > Pipelines > Settings. Disabled the 'Limit job authorization'
  • The trigger above activates on any v0.* or v1.* git tags on the repo. There is no branch filter. Your team will need to careful to tag only after PR or make your own tag strategy.
    • You can not alter the trigger section to include branches:exclude: [], that updates the trigger to also independently watch any non-includes branches for changes, separate from tag triggers.