Fragmented Thought

Creating Azure DevOps repository subscription pipelines

Azure DevOps - Project specific pipeline triggered by repository resource tag
By

Published:

Lance Gliser

Heads up! This content is more than six months old. Take some time to verify everything still works as expected.

Creating pipelines is a funny business. There are several concerns to watch for that would be standard in version control, but require special work instead. I'd like to review the idea of a SemVer'esk major version control system today. I'll be covering Azure DevOps pipelines, specifically, multi-stage > deployment and templates.

That's how I felt almost a year and a half ago. The links to documentation are still useful, but the method was hopelessly complex due requirements we've gotten past. In this post, I'd like to show how easy life can be instead. Today's example will create subscriber pipelines delivering project specific environmental variables to deliver a generic API shipped to client environments.

Creating the generic code base

Examples will be based on an updated version of my previous guide to deploy a Typescript GraphQL API via CDK. There will be some differences, and I'll repost the pipeline specifics here to ensure there's no confusion as our code base has evolved a bit.

We'll be working a project structure that has this organization:

  • azure-pipelines - Validation pipeline for testing and deployment templates
  • bin - CDK project standard
  • lambda - The src code for the lambda API
    • package.json
  • lib - CDK project standard
  • test - CDK project standard
  • package.json

Deployment templates

azure-pipelines/deployment/templates/npm-update.yml

This ensure we're working on the latest version of npm. Hosted machines have an older version by default. This bit us a couple times as the handling of peerDependencies changed since then.

steps: - task: CmdLine@2 inputs: script: sudo npm install npm@latest --location=global displayName: npm install npm@latest --location=global

azure-pipelines/deployment/templates/npm-ci.yml

This installs all the required node modules, both for the root CDK project, and any APIs it deploys.

parameters: # The location of the chart downloaded to the stage from the artifact - name: workingDirectory type: string default: $(Pipeline.Workspace)/drop steps: - task: Npm@1 inputs: command: ci workingDir: ${{ parameters.workingDirectory }} displayName: npm ci - root - task: Npm@1 inputs: command: ci workingDir: ${{ parameters.workingDirectory }}/lambda displayName: npm ci - lambda

azure-pipelines/deployment/templates/aws-deploy.yml

This performs the actual CDK deployment.

parameters: - name: awsCredentials type: string - name: regionName type: string - name: cdkAwsArguments type: string default: "" - name: workingDirectory type: string default: $(Pipeline.Workspace)/drop steps: - script: | echo "Installing packages" sudo npm install --location=global aws-cdk ts-node displayName: "Installing node lambda for CDK" - task: AWSShellScript@1 inputs: awsCredentials: ${{ parameters.awsCredentials }} disableAutoCwd: true inlineScript: cdk deploy --ci --require-approval never ${{ parameters.cdkAwsArguments }} regionName: ${{ parameters.regionName }} scriptType: inline workingDirectory: ${{ parameters.workingDirectory }} displayName: aws cdk deploy

Create a subscriber

Consider maintainability

If you're already thinking in publish and subscription patterns, it makes sense to step back from our immediate work and consider a reliable intermediary. Subscribers will need to maintain .yaml files. That's a painful process over time unless you get out ahead of it. Let's give them an easy method for doing so. In the same project you've created your API, create a new project and we'll encourage subscribers to fork it into their projects to get up to speed, and allow upstream sync to handle any changes you make as the API author.

Writing a repository subscriber pipeline

I wish this were more complicated or impressive, but am so happy that it is not. Unlike the mess with artifact subscriptions, repository subscriptions are dreamy:

azure-pipelines/{RepositoryName}/pipeline.yaml

# Starter pipeline # Start with a minimal pipeline that you can customize to build and deploy your code. # Add steps that build, run tests, deploy, and more: # https://aka.ms/yaml # Do not automatically trigger if definition files update. # That would cause all pipelines in this repo to trigger. trigger: none resources: repositories: - repository: {RepositoryName} # The name used to reference this repository in the checkout step type: git name: {ProjectName}/{RepositoryName} trigger: # CI trigger for this repository, no CI trigger if skipped (only works for Azure Repos) tags: include: - v0.* - v1.* pool: vmImage: ubuntu-latest demands: - npm variables: - name: logLevel value: info - name: subdomain value: auth-api stages: - stage: Dev displayName: Deploy to the Dev environment jobs: - deployment: Deploy environment: {EnvironmentName} # The string name of an Azure DevOps environment variables: # TODO define variable group for AWS variables in the consuming project - group: "" # The name of the variable group - name: awsHostedZoneId value: $[variables.AWS_RT53_HOSTED_ZONE_ID] - name: awsHostedZoneDomain value: $[variables.AWS_RT53_HOSTED_ZONE] - name: awsRegion value: $[variables.AWS_REGION] - name: awsAccessKey value: $[variables.AWS_ACCESS_KEY_ID] - name: awsSecretKey value: $[variables.AWS_SECRET_ACCESS_KEY] strategy: runOnce: deploy: steps: # Because this is a single repository checkout, $(Build.SourcesDirectory) is just this repo. - checkout: {RepositoryName} - template: azure-pipelines/deployment/templates/npm-update.yml@{RepositoryName} - template: azure-pipelines/deployment/templates/npm-ci.yml@{RepositoryName} parameters: workingDirectory: $(Build.SourcesDirectory) - template: azure-pipelines/deployment/templates/aws-deploy.yml@{RepositoryName} parameters: awsCredentials: "" # TODO update to service connection in project cdkAwsArguments: >- --context areAlarmsEnabled=$(areAlarmsEnabled) --context hostedZoneId=$(awsHostedZoneId) --context hostedZoneDomain=$(awsHostedZoneDomain) --context logLevel=$(logLevel) --context subdomain=$(subdomain) regionName: $(awsRegion) workingDirectory: $(Build.SourcesDirectory)

A special note about cdkAwsArguments: The >- yaml concats the lines into a single string for script args. You can't have comments, nor extra lines of white space in it.

Trigger considerations

  • Pipelines can not naturally reach into other projects. You must change a setting:
    Project > Settings > Pipelines > Settings. Disabled the 'Limit job authorization'
  • The trigger above activates on any v0.* or v1.* git tags on the repo. There is no branch filter. Your team will need to careful to tag only after PR or make your own tag strategy.
    • You can not alter the trigger section to include branches:exclude: [], that updates the trigger to also independently watch any non-includes branches for changes, separate from tag triggers.