Skip to main content

Azure DevOps YAML based CD pipeline with multiple artifacts

In my post called "Switch from Azure DevOps release pipeline to YAML based CI/CD pipeline" I explained how to implement a combined CI/CD pipeline based on YAML templates. Because most of the deployments in my daily work are consuming multiple artifacts I had to do some additional R&D to find a solution for those releases in Azure DevOps.

Base structure

In the first post about the transition to YAML CI/CD pipelines I introduced the base structure. The templates I want to reuse are added to the yaml-template folders in the root of my GIT repo. For the CD pipelines which lack a CI phase I introduce a new folder called yaml-releases which will contain the YAML files for releases which are based on multiple artifacts.

The folder structure within my GIT repo is thus:

  • yaml-releases
    • releaseArea
      • myMultipleArtifactsRelease
        • azure-pipelines.yml
        • cd-job.yml
  • yaml-templates
    • jobs
      • ... templates ...
    • stages
      • ... templates ...
    • steps
      • ... templates ...
    • variables
      • ... templates ...

So to recap the templates which can be reused within CI and CD pipelines are located in the yaml-templates folder and the more specific templates are located with the yaml-releases folder.

CD phase

The CD phase of a YAML release which is based on multiple artifacts should be started within an >b>azure-pipelines.yml file. In this file we skip the CI phase because I will be consuming two ARM template artifacts. An ARM template which contains the logic to create a resource group and one template for creating a virtual network. The pipeline will be started manually.

To consume multiple artifacts in a YAML pipeline we have to add a resources object. In this case I reference the artifacts produced by two Azure DevOps CI pipelines. The pipeline is a resource variable identifier which we can use in the other templates and the source is the name of the pipeline that produced the artifact we want to reference. In this case the pipeline is within the same team project so we only have to define the name of the pipeline. You can also reference artifacts produced by pipelines in other team projects, reference repos, containers and also packages. See for all the posibilities the official documentation.

# Build numbering format
name: $(BuildDefinitionName).$(Year:yy)$(DayOfYear)$(rev:.r)

# External references needed by this CI/CD pipeline
resources:

  # References to pipelines that produce artifacts
  pipelines:

    - pipeline: 'ResourceGroups' # Identifier for the resource (used in pipeline resource variables)
      source: 'ResourceGroups-CI' # Name of the pipeline that produces an artifact

    - pipeline: 'VirtualNetworks'
      source: 'VNET-CI'


# Pipeline trigger
trigger: 'none' # Opt out of CI triggers

stages:

# CI phase; this pipeline is only consuming artifacts

# CD phase
- template: ../../../yaml-templates/stages/cd-sandbox-nonprod-prod.yml
  parameters:
    resourceName: '##ENVIRONMENT##-dmz-vnet'
    resourceType: 'Network'
    jobPath: ../../yaml-releases/releaseArea/myMultipleArtifactsRelease/cd-job.yml

Stages

This deployment is using a general stage template located within the yaml-templates folder. The results of a feature branch are deployed in a sandbox environment and only the master branch will be deployed to the Non-Production and Production environment.

This stage template is slightly different than the version used by the combined CI/CD YAML pipeline of my earlier post. This version is modified to provide a link to the correct Job template and contains some modifications for loading environmental information from variable templates.

# stages/cd-sandbox-nonprod-prod.yml

# Core Infra (CD) | Sandbox-Non-Production-Production flow

parameters:

# Name of the resource to deploy
- name: 'resourceName'
  type: 'string'

# Type of resource
- name: 'resourceType'
  type: 'string'

# Location of the job template
- name: 'jobPath'
  type: 'string'

stages:

  # CD phase - deploy only feature branches
  - stage: 'cdSandbox'
    displayName: 'Deploy | Sandbox'
    variables:
    - name: 'isMain'
      value: $[eq(variables['Build.SourceBranch'], 'refs/heads/master')]
    - template: ../variables/deployment-sandbox.yml
    condition: and(succeeded(), eq(variables.isMain, false))
    jobs:
    - template: ${{ parameters.jobPath }}
      parameters:
        resourcePath: ../../../yaml-templates/variables/resources-sandbox.yml
        deployObject:
          environmentName: 'Sandbox'
          environmentAbbreviation: ${{ variables.env }}
          serviceConnectionName: ${{ variables.serviceConnectionName }}
          storageAccountName: ${{ variables.deployStorageAccountName }}
          subscriptionId: ${{ variables.deploySubscriptionId }}
          resourceGroupName: ${{ variables.deployResourceGroupName }}
          location: ${{ variables.location }}
        resourceObject:
          type: ${{ parameters.resourceType }}
          name: ${{ replace(parameters.resourceName, '##ENVIRONMENT##', variables.env) }} # Replace environment placeholder
          environmentType: 'Sandbox'

  # CD phase - deploy only main branch
  - stage: 'cdNonProduction'
    displayName: 'Deploy | Non-Production'
    variables:
    - name: 'isMain'
      value: $[eq(variables['Build.SourceBranch'], 'refs/heads/master')]
    - template: ../variables/deployment-nonprod.yml
    dependsOn: [] # This removes the implicit dependency on previous stage and causes this to run in parallel
    condition: and(succeeded(), eq(variables.isMain, true))
    jobs:
    - template: ${{ parameters.jobPath }}
      parameters:
        resourcePath: ../../../yaml-templates/variables/resources-nonprod.yml
        deployObject:
          environmentName: 'Non-Production'
          environmentAbbreviation: ${{ variables.env }}
          serviceConnectionName: ${{ variables.serviceConnectionName }}
          storageAccountName: ${{ variables.deployStorageAccountName }}
          subscriptionId: ${{ variables.deploySubscriptionId }}
          resourceGroupName: ${{ variables.deployResourceGroupName }}
          location: ${{ variables.location }}
        resourceObject:
          type: ${{ parameters.resourceType }}
          name: ${{ replace(parameters.resourceName, '##ENVIRONMENT##', variables.env) }} # Replace environment placeholder
          environmentType: 'NonProd'

  - stage: 'cdProduction'
    displayName: 'Deploy | Production'
    variables:
    - template: ../variables/deployment-prod.yml
    jobs:
    - template: ${{ parameters.jobPath }}
      parameters:
        resourcePath: ../../../yaml-templates/variables/resources-prod.yml
        deployObject:
          environmentName: 'Production'
          environmentAbbreviation: ${{ variables.env }}
          serviceConnectionName: ${{ variables.serviceConnectionName }}
          storageAccountName: ${{ variables.deployStorageAccountName }}
          subscriptionId: ${{ variables.deploySubscriptionId }}
          resourceGroupName: ${{ variables.deployResourceGroupName }}
          location: ${{ variables.location }}
        resourceObject:
          type: ${{ parameters.resourceType }}
          name: ${{ replace(parameters.resourceName, '##ENVIRONMENT##', variables.env) }} # Replace environment placeholder
          environmentType: 'Prod'

Jobs

The job template is located in the same folder as the azure-pipelines.yml file. This is necessary because this template contains all the logic to deploy the specific set of artifacts referenced by the release.

Within the strategy section of the job we define the steps to execute by this release. First we download the referenced artifacts to the Agent and upload the files to a storage account, than we deploy the resource group, promote the output variable containing the resource group name to a pipeline variable and the last action is deploying the VNET ARM template.

# cd-job.yml

parameters:

# Path to the resource file with environment specific override parameter variables
- name: 'resourcePath'
  type: 'string'

# Object with general deployment information
# deployObject.environmentName
# deployObject.environmentAbbreviation - Abbreviation for the environment (for example sb, np or p)
# deployObject.serviceConnectionName
# deployObject.storageAccountName
# deployObject.subscriptionId
# deployObject.resourceGroupName
# deployObject.location
- name: deployObject
  type: object

# Object with information about the resource that will be deployed
# resourceObject.type - type of resource (for example API, Product or Service)
# resourceObject.name - Name of the resource
# resourceObject.environmentType - Which OTAP environment is it (for example Dev, Test, Acc or Prod)
# resourceObject.resourceGroupName
# resourceObject.deploymentScope - The scope of ARM deployment: Resource Group, Subscription or Management Group
- name: 'resourceObject'
  type: 'object'

jobs:
- deployment: ${{ parameters.resourceObject.type }}_${{ replace(parameters.resourceObject.name, '-', '_') }}_${{ parameters.resourceObject.environmentType }}
  displayName: ${{ parameters.resourceObject.type }} | ${{ parameters.resourceObject.name }}
  variables:
  - template: ../../../yaml-templates/variables/pipeline.yml
  - template: ${{ parameters.resourcePath }}
  pool:
    name: ${{ variables.poolName }}
  environment: ${{ parameters.deployObject.environmentName }}
  strategy:
    runOnce:
      deploy:
        steps:
          - checkout: 'none' # Suppress check out source code

          # Download artifacts from referenced pipelines
          - download: 'ResourceGroups'
            displayName: 'Download artifact - Resource Groups'

          - download: 'VirtualNetworks'
            displayName: 'Download artifact - Virtual Networks'

          # Deploy sequence
          - template: ../../../yaml-templates/steps/cd-copy-all-artifacts.yml
            parameters:
              deployObject: ${{ parameters.deployObject }}

          - template: ../../../yaml-templates/steps/cd-arm-templates.yml
            parameters:
              deployObject: ${{ parameters.deployObject }}
              templateObject:
                displayName: 'Resource Group'
                path: $(Agent.BuildDirectory)/ResourceGroups/templates/resourceGroup.json
                parameterPath: $(Agent.BuildDirectory)/ResourceGroups/templates/parameters/network/resourceGroup.parameters.network-${{ parameters.deployObject.environmentAbbreviation }}.json
                deploymentOutputs: 'resourceGroupName'
              resourceObject:
                deploymentScope: 'Subscription'

          - task: PowerShell@2
            displayName: 'Promote deployment outputs of "resourceGroupName" to pipeline variable'
            inputs:
              targetType: inline
              script: |
                $var=ConvertFrom-Json '$(resourceGroupName)'
                $value=$var.Name.value
                Write-Host "##vso[task.setvariable variable=resourceGroupName;]$value"

          - template: ../../../yaml-templates/steps/cd-arm-templates.yml
            parameters:
              deployObject: ${{ parameters.deployObject }}
              templateObject:
                displayName: 'Virtual Network (VNET)'
                downloadName: 'VirtualNetworks'
                path: $(Agent.BuildDirectory)/VirtualNetworks/templates/vnet.orchestrator.json
                parameterPath: $(Agent.BuildDirectory)/VirtualNetworks/templates/parameters/vnet.parameters.dmz-${{ parameters.deployObject.environmentAbbreviation }}.json
                overrideParameters: -logAnalyticsWorkspaceId "$(logAnalyticsWorkspaceId)"
              resourceObject:
                resourceGroupName: $(resourceGroupName)
                deploymentScope: 'Resource Group'

Steps

To be able to deploy the nested ARM templates all the artifacts first have to be uploaded to a storage account. In the Azure File Copy task we use a wildcard copy of all the artifacts which where earlier copied by the download task to the build directory.

# steps/cd-copy-all-artifacts.yml

# Copy all artifacts to storage account in Azure

parameters:

# Object with general deployment information
# deployObject.environmentName
# deployObject.serviceConnectionName
# deployObject.storageAccountName
# deployObject.subscriptionId
# deployObject.resourceGroupName
# deployObject.location
- name: deployObject
  type: object

steps:

  - task: AzureFileCopy@4
    name: 'deploy' # The name used for the output variables
    displayName: 'Copy all artifacts to storage account'
    inputs:
      sourcePath: '$(Agent.BuildDirectory)/*' # All the artifacts are downloaded to the workspace directory for a particular pipeline
      azureSubscription: ${{ parameters.deployObject.serviceConnectionName }}
      destination: 'AzureBlob'
      storage: ${{ parameters.deployObject.storageAccountName }}
      containerName: 'deploy'
      blobPrefix: $(Build.BuildNumber)-$(System.StageName)

The ARM templates deployment task for this release pipeline is a slightly modified version used in the earlier combined CI/CD pipeline. The copy task is removed and the displayName is different this time because I want to output the functional name of the ARM template which will be deployed.

# steps/cd-arm-templates.yml

# Deploy the ARM template to Azure

parameters:

# Object with general deployment information
# deployObject.environmentName
# deployObject.serviceConnectionName
# deployObject.storageAccountName
# deployObject.subscriptionId
# deployObject.resourceGroupName
# deployObject.location
- name: deployObject
  type: object

# templateObject.displayName - Name to display in the agent log
# templateObject.downloadName - Name of the download reference
# templateObject.path - Path or pattern pointing to the ARM template
# templateObject.parameterPath - Path or pattern pointing to the parameters file for the ARM template
# templateObject.overrideParameters - Additional override parameters (besides storageAccount and storageAccountSASToken)
# templateObject.deploymentOutputs - Provide a name for the variable for the output variable which will contain the outputs section of the current deployment object in string format
- name: templateObject
  type: object

# Object with information about the resource that will be deployed
# resourceObject.type - type of resource (for example API, Product or Service)
# resourceObject.name - Name of the resource
# resourceObject.environmentType - Which OTAP environment is it (for example Dev, Test, Acc or Prod)
# resourceObject.resourceGroupName
# resourceObject.deploymentScope - The scope of ARM deployment: Resource Group, Subscription or Management Group
- name: resourceObject
  type: object

steps:

  - task: AzureResourceManagerTemplateDeployment@3
    displayName: 'Deploy ARM template - ${{ parameters.templateObject.displayName }}'
    inputs:
      deploymentScope: ${{ parameters.resourceObject.deploymentScope }}
      azureResourceManagerConnection: ${{ parameters.deployObject.serviceConnectionName }}
      subscriptionId: ${{ parameters.deployObject.subscriptionId }}
      action: 'Create Or Update Resource Group'
      # Add for resource group scoped deployments
      ${{ if eq(parameters.resourceObject.deploymentScope, 'Resource Group') }}:
        resourceGroupName: ${{ parameters.resourceObject.resourceGroupName }}
      location: ${{ parameters.deployObject.location }}
      csmFile: ${{ parameters.templateObject.path }}
      csmParametersFile: ${{ parameters.templateObject.parameterPath }}
      overrideParameters: -cdStorageAccount "$(deploy.StorageContainerUri)$(Build.BuildNumber)-$(System.StageName)/${{ parameters.templateObject.downloadName }}/templates" -cdStorageAccountSASToken "$(deploy.StorageContainerSasToken)" ${{ parameters.templateObject.overrideParameters }}
      deploymentMode: 'Incremental'
      deploymentOutputs: ${{ parameters.templateObject.deploymentOutputs }}

These are the template changes I had to make to do a deploy of multiple artifacts in a CD pipeline based on YAML.

Labels:

Comments

Post a Comment

Popular posts from this blog

CS8357: The specified version string contains wildcards, which are not compatible with determinism.

Today I was busy with creating a WCF service solution in Visual Studio Enterprise 2017 (15.9.2). In this solution I use a few C# class libraries based on .NET 4.7.2. When I compiled the solution I got this error message: Error CS8357: The specified version string contains wildcards, which are not compatible with determinism. Either remove wildcards from the version string, or disable determinism for this compilation The error message is linking to my AssemblyInfo.cs file of the Class library projects. In all the projects of this solution I use the wildcard notation for generating build and revision numbers. // Version information for an assembly consists of the following four values: // // Major Version // Minor Version // Build Number // Revision // // You can specify all the values or you can default the Build and Revision Numbers // by using the '*' as shown below: // [assembly: AssemblyVersion("1.0.*")] [assembly: AssemblyVersion("1.0.
Post a Comment
Read more

Fixing HTTP Error 401.2 unauthorized on local IIS

Sometimes the Windows Authentication got broken on IIS servers so you cannot log in locally on the server. In that case you get the dreadfully error message HTTP Error 401.2 - Unauthorized You are not authorized to view this page due to invalid authentication headers. To fix this issue you can repair the Windows Authentication feature with the following PowerShell commands: Remove-WindowsFeature Web-Windows-Auth Add-WindowsFeature Web-Windows-Auth
Post a Comment
Read more

Make steps conditional in multi-stage YAML pipelines

To make the switch from the graphical release pipelines in Azure DevOps I am missing two features. The first one is to be able to defer a deploy and the second one is to exclude certain deployment steps without the need for editing the YAML file.  The defer option is something Microsoft has to solve in their Azure DevOps proposition. It's a feature which you have in the graphical release pipeline but what they have not implemented yet in their YAML pipeline replacement. Approvals and certain gate conditions are implemented on the environment but the defer option is still missing .  Pipeline The conditional deployment option can be implemented with the help of runtime parameters and expressions . In the parameter section you define boolean parameters which will control the deploy behavior. With the expressions you can control which stage/job/task should be executed when the pipeline runs. In the below YAML sample I experimented with conditions in the azure-pipelines.yml  file
Post a Comment
Read more