It has been years ago I started applying continuous integration (CI) and continuous deployment (CD) principles at work. Around 2005, it was the time of Visual Studio 2005 Team System, we started using a build server in our daily development process. From then on it was no longer more: "it works on my machine". With every check-in of code changes a CI build was started which compiled your code and run unit tests which verified the quality of your code.
Deployment was in the beginning still a process of manually copying build artifacts and script files to servers and doing the installation process based on the instructions from the deployment manual (if available). Gradually it migrated to scripts (mix of batch/PowerShell/VBScript files) that our release automatically ran from the build server.
In 2016 we migrated from our on-premise TFS 2008 based environment to the cloud solution nowadays known as Azure DevOps. We upgraded our MSBuild driven CI/CD pipelines to the new visual CI pipelines and adopted also the visual release pipelines. It was a major step ahead from the code version we used earlier. We use variable groups and task groups heavenly to make our life easier.
With the growing number of pipelines, the pain of maintaining all those visual pipelines becomes apparent. Refactoring is going to be a time consuming process and so is the lack of full values source control. In terms of source control, we follow the market with the adoption of GIT for our infrastructure as code projects and our iPaaS code. With the adoption of GIT we also started with YAML based CI pipelines. Our releases are still based on release pipelines and we are missing the efficiencies of code based releases pipelines.
Microsoft added the last year YAML based CD functionality in their pipeline for Azure DevOps. The last few weeks I invested some time in learning to use YAML for a combined CI/CD pipeline. My R&D case was building a pipeline for deploying APIs and products in API Management based on ARM templates. The results of this R&D quest you can find in this post.
Reuse
To maximize reuse within the YAML pipeline I adopted the usage of templates. Templates are available on four levels: stages, jobs, steps and variables and propagate reuse. Like C# code you pass information between templates by using parameters. For a complete overview of the possibilities read the YAML schema reference page.
Base structure
To propagate reuse I added a yaml-templates folder to the root of my GIT repo. Within that folder I made a folder per template type.
- jobs
- stages
- steps
- variables
In each folder you add the YAML file with a template piece you can reuse. For clarity I use a prefix which indicate the usage. The CI templates are prefixed with ci- and the CD templates are prefixed with cd-.
Pipeline
The starting point is the pipeline which references the azure-pipelines.yml file. A naming convention introduced by Azure DevOps itself. This file contains the settings for the build numbering format, the pipeline trigger for starting automatically the pipeline and references the stages templates for the CI and CD phase.
In the example below you find the definition I use for deploying an API to API Management. Because the templates I wrote propagate reuse I use a boolean parameter value to indicate that this pipeline is used for an API CI/CD run. For the rest I pass some information about the folder where the templates reside in and for the CD phase also the path to the (nested) template and the parameter files with the environmental settings.
# Build numbering format name: $(BuildDefinitionName).$(Year:yy)$(DayOfYear)$(rev:.r) # Pipeline trigger trigger: batch: 'true' branches: include: - 'refs/heads/*' paths: include: - 'apis/myApi/*' stages: # CI phase - template: ../../yaml-templates/stages/ci-arm-templates.yml parameters: armPath: 'apis/myApi/' isApi: true # CD phase - template: ../../yaml-templates/stages/cd-apim.yml parameters: resourceName: 'MyApi' resourceType: 'API' templateObject: path: $(Agent.BuildDirectory)\templates\nestedTemplate.json parameterPath: $(Agent.BuildDirectory)\templates\parameters\myApi.parameters.apim-##ENVIRONMENT##.json
CI phase
Our pattern for ARM templates is that in the CI phase we do a JSON syntax check with JSONLint to identify obvious mistakes like missing commas and other formatting issues. After that we copy the templates to a storage account (needed for nested templates) and do a verification deploy. If no validation errors are detected we add the templates as an artifact file to the pipeline so we are sure which version of the templates we will deploy in the CD phase.
In template form we get the following structure of YAML files:
- stage: ci-arm-templates.yml
- job: ci-arm-templates.yml
- step: ci-json-validation.yml
- step: ci-arm-validation.yml
- step: ci-publish-artifact.yml
In these files we reference variable templates where we stored environment specific settings.
Stages
In the stages template I am preparing the actions needed in the CI phase of the pipeline. Because the CI phase is quite general the template is actual only passing through the parameter information to the actual jobs template and is giving some documentation about the object parameters. The stages template for the CD phase is more intelligent.
# stages/ci-arm-templates.yml # Contains all the actions which are needed to do the CI phase for ARM templates parameters: # The folder where the ARM file(s) reside in - name: 'armPath' type: string # Scope of deployment - name: 'deploymentScope' type: string values: - Resource Group - Subscription - Management Group default: Resource Group # templateObject.path - Path or pattern pointing to the ARM template # templateObject.parameterPath - Path or pattern pointing to the parameters file for the ARM template # templateObject.overrideParameters - Additional override parameters (besides storageAccount and storageAccountSASToken) - name: templateObject type: object default: # Set to true when it is an API build. - name: isApi type: boolean default: false stages: # CI phase - stage: 'ci' displayName: 'Build' jobs: # Validate & package ARM template(s) - template: ../jobs/ci-arm-templates.yml parameters: armPath: ${{ parameters.armPath }} deploymentScope: ${{ parameters.deploymentScope }} templateObject: ${{ parameters.templateObject }} isApi: ${{ parameters.isApi }}
Jobs
The jobs template is the place where we reference the three steps of our CI phase: the template for the JSON validation, the template for the validation deploy and the template which contains the publishing step. We reference also two variable files. One with the information about the name of the Azure DevOps agent we want to use and one with information about the Azure Blob Storage account.
This template also use the concept of conditional insertions. For APIs I have to reference a parameter file. Because we have a naming convention I can hard code the name and location of that file relative within the folder which is defined in the armPath parameter. For all other CI runs we use the information defined in the templateObject parameter.
# jobs/ci-arm-templates.yml # Contains all the jobs needed for controlling that the ARM templates have a valid syntax and are # deployable to an Azure subscription. The last step is deploying the ARM templates as an artifact # to the release pipeline. parameters: # The folder where the ARM file(s) reside in - name: 'armPath' type: string # Scope of deployment - name: 'deploymentScope' type: string # templateObject.path - Path or pattern pointing to the ARM template # templateObject.parameterPath - Path or pattern pointing to the parameters file for the ARM template # templateObject.overrideParameters - Additional override parameters (besides storageAccount and storageAccountSASToken) - name: 'templateObject' type: object # Set to true when it is an API build. - name: 'isApi' type: boolean jobs: - job: 'validateAndPackage' displayName: 'Validate & package ARM templates' variables: - template: ../variables/pipeline.yml - template: ../variables/azure-deployment-np.yml pool: name: ${{ variables.poolName }} steps: - template: ../steps/ci-json-validation.yml parameters: jsonPath: ${{ parameters.armPath }} - template: ../steps/ci-arm-validation.yml parameters: armPath: ${{ parameters.armPath }} # If it is an API CI run calculate the path to the correct templates ${{ if eq(parameters.isApi, true) }}: templateObject: path: ${{ parameters.armPath }}/orchestrator.json parameterPath: ${{ parameters.armPath }}/orchestrator.parameters.json # Use the supplied parameter value for non API related CI runs ${{ if eq(parameters.isApi, false) }}: templateObject: ${{ parameters.templateObject }} deployObject: serviceConnectionName: ${{ variables.serviceConnectionName }} storageAccountName: ${{ variables.deployStorageAccountName }} subscriptionId: ${{ variables.deploySubscriptionId }} resourceGroupName: ${{ variables.deployResourceGroupName }} location: ${{ variables.location }} deploymentScope: ${{ parameters.deploymentScope }} - template: ../steps/ci-publish-artifact.yml parameters: artifactPath: ${{ parameters.armPath }} artifactName: 'templates'
Steps
The steps templates reference the Azure Devops tasks we need to do the actual work. The first template defines the steps we need to do the JSON validation with JSONLint. We have to install Node.js and JSONLint package first for we can do the actual validation.
# steps/ci-json-validation.yml # Validates all the JSON files within the specified path (directory) with the tool JSONLint # See: https://www.npmjs.com/package/jsonlint parameters: - name: 'jsonPath' type: string steps: - task: UseNode@1 displayName: 'Install Node.js' inputs: checkLatest: true - task: Npm@1 displayName: 'Install JSONLint' inputs: command: 'custom' customCommand: 'install jsonlint -g' - task: CmdLine@2 displayName: 'Validate Syntax JSON file(s)' inputs: script: 'FOR /r %%A IN (*.json) DO jsonlint %%A' workingDirectory: ${{ parameters.jsonPath }} failOnStderr: true
In the ARM validation step we copy all the templates to a Azure Blob Storage account. This is needed when you do a validation run of a nested ARM template because the pipeline needs access to the files. In the storage account we use two containers: one for the build phase and one for the deploy phase. A lifecycle policy will automatically delete files older than one day so we optimize our costs. With conditional insertions we handle the differences between resource group and subscription deployment ARM templates.
# steps/ci-arm-validation.yml # Validates that the ARM template is deployable to Azure parameters: # The folder where the ARM file(s) reside in - name: 'armPath' type: string # Object with general deployment information # deployObject.environmentName # deployObject.serviceConnectionName # deployObject.storageAccountName # deployObject.subscriptionId # deployObject.resourceGroupName # deployObject.location # deployObject.deploymentScope - The scope of ARM deployment: Resource Group, Subscription or Management Group - name: 'deployObject' type: object # templateObject.path - Path or pattern pointing to the ARM template # templateObject.parameterPath - Path or pattern pointing to the parameters file for the ARM template # templateObject.overrideParameters - Additional override parameters (besides storageAccount and storageAccountSASToken) - name: 'templateObject' type: object steps: - task: AzureFileCopy@4 name: 'deploy' # The name used for the output variables displayName: 'Copy ARM template(s)' inputs: sourcePath: '$(System.DefaultWorkingDirectory)/${{ parameters.armPath }}/*' azureSubscription: ${{ parameters.deployObject.serviceConnectionName }} destination: 'AzureBlob' storage: ${{ parameters.deployObject.storageAccountName }} containerName: 'build' blobPrefix: $(Build.BuildNumber) - task: AzureResourceManagerTemplateDeployment@3 displayName: 'Validate ARM template(s)' inputs: deploymentScope: ${{ parameters.deployObject.deploymentScope }} azureResourceManagerConnection: ${{ parameters.deployObject.serviceConnectionName }} subscriptionId: ${{ parameters.deployObject.subscriptionId }} action: 'Create Or Update Resource Group' # Add for resource group scoped deployments ${{ if eq(parameters.deployObject.deploymentScope, 'Resource Group') }}: resourceGroupName: ${{ parameters.deployObject.resourceGroupName }} location: ${{ parameters.deployObject.location }} csmFile: ${{ parameters.templateObject.path }} csmParametersFile: ${{ parameters.templateObject.parameterPath }} overrideParameters: -storageAccount "$(deploy.StorageContainerUri)$(Build.BuildNumber)" -storageAccountSASToken "$(deploy.StorageContainerSasToken)" ${{ parameters.templateObject.overrideParameters }} deploymentMode: 'Validation'
The last step will publish the ARM templates in ZIP format to the pipeline. We could also reference the actual files in the GIT repo when doing the CD phase but by publishing the ARM templates as an artifact to the pipeline I am sure which version of the templates are used: the verified ones when we did the CI phase.
# steps/ci-publish-artifact.yml # Publish artifacts to the pipeline parameters: # The folder where the files reside in - name: 'artifactPath' type: string # The name of the artifact which will be published to the pipeline - name: 'artifactName' type: string steps: - task: PublishPipelineArtifact@1 displayName: 'Publish Pipeline Artifact:' ${{ parameters.artifactName }} inputs: targetPath: ${{ parameters.artifactPath }} artifact: ${{ parameters.artifactName }}
Variables
The variables templates contains the same information I normally store in the Azure DevOps library as variable groups. In this case we have the advantage of source control so this information is versioned. Security related information as passwords for example you don't store in variable templates. This kind of information you have to store in Azure Key Vault for example and reference that information from the ARM templates directly or pass in as parameters to your scripts as secure strings. You can also use the Azure DevOps variable templates and reference the Azure Key Vault values you need as variables.
The first variable template is quit simple. It only contains Azure DevOps pipeline information and that is in this case only the name of the Default pool. We host our own pool agents on Azure Virtual Machines because of our requirement to control the software on the agent and because of the line of sight to components within our environment for deployments.
# variables/pipeline.yml # Contains variables specific for Azure DevOps pipelines variables: - name: 'poolName' value: Default
The second variable template contains information about the Azure Blob Storage account we use for storing the nested templates for our CI and CD stages.
# variables/azure-deployment-np.yml # Contains variables with Azure environment information for non-production deployments variables: - name: 'serviceConnectionName' value: 'azureSubscription-apim-nonProduction' - name: 'location' value: 'West Europe' - name: 'deployResourceGroupName' value: 'ci-cd-rg' - name: 'deployStorageAccountName' value: 'apimcicdnp' - name: 'deploySubscriptionId' value: '00000000-0000-0000-0000-000000000000'
CD phase
The actual deployment of the API Management ARM templates is done in the CD phase. In this case we only need the artifacts from a single CI phase so we can combine the CI and CD phase in one azure-devops.yml file and the only action that has to be done is copy the artifact content to the Azure Blob Storage account and do the actual deployment.
In template form we get the following structure of YAML files:
- stage: cd-apim.yml
- job: cd-arm-templates.yml
- step: cd-arm-templates.yml
In these files we also reference variable templates where we stored environment specific settings.
Stages
The stage template for the CD phase is more complex than the one you will need for the CI phase. The stage template has to define every environment (stage) where the ARM templates have to deployed to. In this example the environment is split up in three API Management instances. One called staging where we do trial deployment of the ARM templates and a non-production and production environment.
For YAML template based CD phase where you need manual approvals you will need an Azure DevOps environment. An environment is a collection of resources, such as Kubernetes clusters and virtual machines, that can be targeted by deployments from a pipeline. On the environment you can configure for example approvals.
In the stage template you will find quit what references to environmental variable templates. Also the parameters are configurered with references to ARM template parameter files. By using a place holder text in the azure-pipelines.yml and a naming convention for the actual parameter files we can reference the correct file in each stage. The usage of conditions made it possible to control when deployments to stages may take place. The staging environment will only accept deployments for manual runs of feature branches. Deployments to the non-production environment are started automatically when the master/main CI build as taken place.
# stages/cd-apim.yml # APIM pipeline (CD) | Staging-NonProduction-Production flow # Name of the resource to deploy parameters: - name: 'resourceName' type: string # Type of resource - name: 'resourceType' type: string values: - Service # Scope of deployment - name: 'deploymentScope' type: string values: - Resource Group - Subscription - Management Group default: Resource Group # templateObject.path - Path or pattern pointing to the ARM template # templateObject.parameterPath - Path or pattern pointing to the parameters file for the ARM template # templateObject.overrideParameters - Additional override parameters (besides storageAccount and storageAccountSASToken) - name: 'templateObject' type: object stages: # CD phase - deploy only feature branches - stage: 'cdStaging' displayName: 'Deploy | Staging' variables: - name: 'isMain' value: $[eq(variables['Build.SourceBranch'], 'refs/heads/master')] - template: ..\variables\azure-deployment-np.yml - template: ..\variables\apim-staging.yml dependsOn: ci # Start only for manual feature builds (prevent that CD phase is started always, for example for merges) condition: and(succeeded(), eq(variables.isMain, false), eq(variables['Build.Reason'], 'Manual')) jobs: - template: ../jobs/cd-arm-templates.yml parameters: deployObject: environmentName: APIM-${{ variables.serviceName }} serviceConnectionName: ${{ variables.serviceConnectionName }} storageAccountName: ${{ variables.deployStorageAccountName }} subscriptionId: ${{ variables.deploySubscriptionId }} resourceGroupName: ${{ variables.deployResourceGroupName }} location: ${{ variables.location }} deploymentScope: ${{ parameters.deploymentScope }} templateObject: path: ${{ parameters.templateObject.path }} parameterPath: ${{ replace(parameters.templateObject.parameterPath, '##ENVIRONMENT##', 'staging') }} # Replace environment placeholder overrideParameters: ${{ parameters.templateObject.overrideParameters }} resourceObject: type: ${{ parameters.resourceType }} name: ${{ parameters.resourceName }} environmentType: 'CD_Staging' resourceGroupName: ${{ variables.resourceGroupName }} # CD phase - deploy only main branch - stage: 'cdNonProd' displayName: 'Deploy | Non-Production' variables: - name: 'isMain' value: $[eq(variables['Build.SourceBranch'], 'refs/heads/master')] - template: ..\variables\azure-deployment-np.yml - template: ..\variables\apim-np.yml dependsOn: 'ci' condition: and(succeeded(), eq(variables.isMain, true)) jobs: - template: ../jobs/cd-arm-templates.yml parameters: deployObject: environmentName: APIM-${{ variables.serviceName }} serviceConnectionName: ${{ variables.serviceConnectionName }} storageAccountName: ${{ variables.deployStorageAccountName }} subscriptionId: ${{ variables.deploySubscriptionId }} resourceGroupName: ${{ variables.deployResourceGroupName }} location: ${{ variables.location }} deploymentScope: ${{ parameters.deploymentScope }} templateObject: path: ${{ parameters.templateObject.path }} parameterPath: ${{ replace(parameters.templateObject.parameterPath, '##ENVIRONMENT##', 'nonprod') }} # Replace environment placeholder overrideParameters: ${{ parameters.templateObject.overrideParameters }} resourceObject: type: ${{ parameters.resourceType }} name: ${{ parameters.resourceName }} environmentType: 'NonProd' resourceGroupName: ${{ variables.resourceGroupName }} - stage: 'cdProd' displayName: 'Deploy | Production' variables: - template: ..\variables\azure-deployment-p.yml - template: ..\variables\apim-p.yml condition: succeeded() jobs: - template: ../jobs/cd-arm-templates.yml parameters: deployObject: environmentName: APIM-${{ variables.serviceName }} serviceConnectionName: ${{ variables.serviceConnectionName }} storageAccountName: ${{ variables.deployStorageAccountName }} subscriptionId: ${{ variables.deploySubscriptionId }} resourceGroupName: ${{ variables.deployResourceGroupName }} location: ${{ variables.location }} deploymentScope: ${{ parameters.deploymentScope }} templateObject: path: ${{ parameters.templateObject.path }} parameterPath: ${{ replace(parameters.templateObject.parameterPath, '##ENVIRONMENT##', 'prod') }} # Replace environment placeholder overrideParameters: ${{ parameters.templateObject.overrideParameters }} resourceObject: type: ${{ parameters.resourceType }} name: ${{ parameters.resourceName }} environmentType: 'Prod' resourceGroupName: ${{ variables.resourceGroupName }}
Jobs
The jobs template for the CD phase defines the deployment strategy and references the correct Azure DevOps environment. For API Management the default run once strategy is used. Based on the information in the resourceObject parameter the deployment and displayName used by the environment are composed. In this case what kind of API Management component (API/Product/Service) we are deploying and the name of that component.
# jobs/cd-arm-templates.yml # Deploy nested ARM templates based on parameter files parameters: # Object with general deployment information # deployObject.environmentName # deployObject.serviceConnectionName # deployObject.storageAccountName # deployObject.subscriptionId # deployObject.resourceGroupName # deployObject.location # deployObject.deploymentScope - The scope of ARM deployment: Resource Group, Subscription or Management Group - name: 'deployObject' type: object # templateObject.path - Path or pattern pointing to the ARM template # templateObject.parameters - Path or pattern pointing to the parameters file for the ARM template # templateObject.overrideParameters - Additional override parameters (besides storageAccount and storageAccountSASToken) - name: 'templateObject' type: object # Object with information about the resource that will be deployed # resourceObject.type - type of resource (for example API, Product or Service) # resourceObject.name - Name of the resource # resourceObject.environmentType - Which OTAP environment is it (for example Dev, Test, Acc or Prod) # resourceObject.resourceGroupName - name: 'resourceObject' type: object jobs: - deployment: ${{ parameters.resourceObject.type }}_${{ parameters.resourceObject.name }}_${{ parameters.resourceObject.environmentType }} displayName: ${{ parameters.resourceObject.type }} | ${{ parameters.resourceObject.name }} variables: - template: ../variables/pipeline.yml pool: name: ${{ variables.poolName }} environment: ${{ parameters.deployObject.environmentName }} strategy: runOnce: deploy: steps: - template: ../steps/cd-arm-templates.yml parameters: deployObject: ${{ parameters.deployObject }} resourceObject: ${{ parameters.resourceObject }} templateObject: ${{ parameters.templateObject }}
Steps
The deployment step is almost the same as the verification step in the CI phase. I could combine both separate templates to one template but I decided to keep them separate for clarity.
# steps/cd-arm-templates.yml # Deploy the ARM template to Azure parameters: # Object with general deployment information # deployObject.environmentName # deployObject.serviceConnectionName # deployObject.storageAccountName # deployObject.subscriptionId # deployObject.resourceGroupName # deployObject.location # deployObject.deploymentScope - The scope of ARM deployment: Resource Group, Subscription or Management Group - name: 'deployObject' type: object # templateObject.path - Path or pattern pointing to the ARM template # templateObject.parameterPath - Path or pattern pointing to the parameters file for the ARM template # templateObject.overrideParameters - Additional override parameters (besides storageAccount and storageAccountSASToken) - name: 'templateObject' type: object # Object with information about the resource that will be deployed # resourceObject.type - type of resource (for example API, Product or Service) # resourceObject.name - Name of the resource # resourceObject.environmentType - Which OTAP environment is it (for example Dev, Test, Acc or Prod) # resourceObject.resourceGroupName - name: 'resourceObject' type: object steps: - task: AzureFileCopy@4 name: 'deploy' # The name used for the output variables displayName: 'Copy ARM template(s)' inputs: sourcePath: '$(Agent.BuildDirectory)/templates/*' azureSubscription: ${{ parameters.deployObject.serviceConnectionName }} destination: 'AzureBlob' storage: ${{ parameters.deployObject.storageAccountName }} containerName: 'deploy' blobPrefix: $(Build.BuildNumber)-$(System.StageName) - task: AzureResourceManagerTemplateDeployment@3 displayName: 'Deploy ARM template(s)' inputs: deploymentScope: ${{ parameters.deployObject.deploymentScope }} azureResourceManagerConnection: ${{ parameters.deployObject.serviceConnectionName }} subscriptionId: ${{ parameters.deployObject.subscriptionId }} action: 'Create Or Update Resource Group' # Add for resource group scoped deployments ${{ if eq(parameters.deployObject.deploymentScope, 'Resource Group') }}: resourceGroupName: ${{ parameters.resourceObject.resourceGroupName }} location: ${{ parameters.deployObject.location }} csmFile: ${{ parameters.templateObject.path }} csmParametersFile: ${{ parameters.templateObject.parameterPath }} overrideParameters: -storageAccount "$(deploy.StorageContainerUri)$(Build.BuildNumber)-$(System.StageName)" -storageAccountSASToken "$(deploy.StorageContainerSasToken)" ${{ parameters.templateObject.overrideParameters }} deploymentMode: Incremental
Variables
I use variable files to store the non-secret environmental information in GIT. In this case the name of the API Management instance and the name of the resource group.
# variables/apim-np.yml # Contains variables with API Management information for non-production deployments variables: - name: 'serviceName' value: 'my-np-apim' - name: 'resourceGroupName' value: 'apim-np-rg'
These are the combined CI/CD YAML templates I created based on my R&D activities for single artifact based deployments for an API Management environment. With a few adaptions the templates are also usable for deployments of other kind of ARM resources.
Comments
Post a Comment