TEKNOLOGI

I recently discovered the extends functionality for Azure DevOps pipelines, which makes it possible to create a pipeline that inherits from another pipeline or template. This feature makes it possible to reuse and extend existing pipeline definitions, which promotes consistency and reducing duplication across CI/CD processes.

I was searching for a way to dynamically run different Azure DevOps pipelines for multiple AKS clusters. As an example I have two processes: bootstrapping an AKS cluster and onboarding workloads onto those clusters. Traditionally, this would require 2 separate pipelines, each defining parameter values for the cluster instances to dynamically run the pipelines. This approach leads to the need to define parameter values for the cluster instances in two different places, which feels reduntant but also increases risk of errors due to duplicate code.

So what happens when a new cluster is provisioned, how can I bootstrap the cluster and onboard the workloads without defining the cluster instance at multiple places? And how can I make it easier to add more processes for the AKS clusters?

This is where the extends functionality comes into play. Instead of defining the cluster instances for each pipeline definition, the pipelines can extend a ‘cluster ochestrator’ pipeline, where the cluster instances are defined. Here’s how it works:

• The pipeline can inherit stages, jobs, steps, variables, and other configurations from a base template.
• The inherited pipeline can be customized by adding or overriding specific stages, jobs, steps, or variables, while still leveraging the common logic defines in the base template.
• The inherited pipeline adds modularity to existing pipelines that can be reused accross templates, projects or repositories.
• Changes made to the inherited base template are automatically propagated to all pipelines that extend it, making it easier to maintain and update the CI/CD processes.

For example, I have a bootstrap pipeline with a stage template to run for each cluster. It extends the cluster orchestrator baseline template to leverage it’s dynamic and reusable structure:

trigger: none

name: 'Cluster bootstrap'

parameters:
- name: PipelineSpecificParameter1
  displayName: 'Pipeline Specific Parameter 1'
  type: boolean
  default: true
- name: PipelineSpecificParameter2
  displayName: 'Pipeline Specific Parameter 2'
  type: string

variables:
  - name: tenant
    value: 'teknologi'

extends:
  template: ./clusters-orchestrator.yml
  parameters:
    stageTemplate: "../templates/stages/aks-bootstrap-cluster.yml"
    additionalParameters:
      PipelineSpecificParameter1: ${{ parameters.PipelineSpecificParameter1 }}
      PipelineSpecificParameter2: ${{ parameters.PipelineSpecificParameter2 }}
      tenantConfigFilePath: '$(Build.SourcesDirectory)/.azuredevops/configs/${{ variables.tenant }}.json'

The pipeline contains some pipeline specific parameters, which are added as additional parameters to the cluster orchestrator pipeline. The orchestrator pipeline also has a parameter for the stage template, which will be invoked in the cluster orchestrator and defines the steps for bootstrapping an AKS cluster.

The cluster orchestrator pipeline definitions contains the cluster instances defined in the tenants parameter.

parameters:
  - name: tenants
    type: object
    default:
      - tenant: "teknologi"
        environments:
          - environmentCode: "dev"
            regions:
              - name: "region1"
                clusters:
                  - "aks01"
              - name: "region2"
                clusters:
                  - "aks01"
                  - "aks02"
          - environmentCode: "tst"
            regions:
              - name: "region1"
                clusters:
                  - "aks01"
              - name: "region2"
                clusters:
                  - "aks01"
                  - "aks02"
          - environmentCode: "prd"
            regions:
              - name: "region1"
                clusters:
                  - "aks01"
              - name: "region2"
                clusters:
                  - "aks01"
                  - "aks02"
  - name: deploymentTenant
    type: string
  - name: stageTemplate
    type: string
    default: ''
  - name: additionalParameters
    type: object
    default: {}

stages:
- ${{ each tenant in parameters.tenants }}:
  - ${{ if eq(tenant.tenant, parameters.deploymentTenant) }}:
    - ${{ each env in tenant.environments }}:
      - ${{ each region in env.regions }}:
        - ${{ each cluster in region.clusters }}:
            - template: ${{ parameters.stageTemplate }}
              parameters:
                stageName: "${{ upper(tenant.tenant) }}_${{ upper(region.name) }}_${{ upper(env.environmentCode) }}_${{ upper(cluster) }}"
                stageCondition: "and(succeeded(), or(eq('${{ env.environmentCode }}', 'dev'), and(eq('${{ env.environmentCode }}', 'tst'), eq(variables['Build.SourceBranch'], 'refs/heads/main')), and(eq('${{ env.environmentCode }}', 'prd'), eq(variables['Build.SourceBranch'], 'refs/heads/main'))))"
                ${{ if eq(variables['Build.SourceBranchName'], 'main') }}:
                  serviceConnection: 'SE-${{ upper(tenant.tenant) }}-${{ upper(env.environmentCode) }}'
                  workloadServiceConnection: 'SERVICECONNECTION-${{ upper(tenant.tenant) }}-${{ upper(env.environmentCode) }}'
                ${{ if ne(variables['Build.SourceBranchName'], 'main') }}:
                  serviceConnection: 'SERVICECONNECTION-${{ upper(tenant.tenant) }}-${{ upper(env.environmentCode) }}'
                  workloadServiceConnection: 'SERVICECONNECTION-${{ upper(tenant.tenant) }}-${{ upper(env.environmentCode) }}'
                poolName: '${{ upper(tenant.tenant) }} ${{ upper(env.environmentCode) }}'
                tenant: ${{ lower(tenant.tenant) }}
                region: ${{ region.name }}
                environment: "AKS-${{ upper(tenant.tenant) }}-${{ upper(env.environmentCode) }}"
                environmentCode: ${{ env.environmentCode }}
                clusterInstanceName: ${{ cluster }}
                tenantConfigFilePath: $(Build.SourcesDirectory)/.azuredevops/configs/${{ tenant.tenant }}.json
                ${{ each pair in parameters.additionalParameters }}:
                  ${{ pair.key }}: ${{ pair.value }}

This reusable Azure DevOps pipeline template is designed to dynamically generate stages and run pipelines for multiple tenants, environments, regions and clusters:

• It iterates over the tenant parameter to generate stages for the specified tenant, environment, region and cluster.
• It only generates stages for the deploymentTenant specified in the pipeline.
• The stageTemplate parameter contains the path to the reusable stage template.
• A stageCondition is added to ensure that stages are only executed under specific conditions, in this case: always run for dev, and only for tst and prd if source branch is main.
• The additionalParameters parameter allows passing custom key-value pairs for specific stage template. For example an custom parameter that is necessary for the bootstrap pipeline template, but not necesarry for the workload onboarding template.

The base template dynamically generates stages for each tenant, environment, region and cluster and uses the stageTemplate to define the steps for each stage. The parameters defined for the stage template are the standard parameters that must be present in every stage template, with the possibility to add additional custom parameters with the additionalParameters parameter.

The purpose for this template is to:

• Standardize the deployment process across multiple tenants, environments, regions and clusters.
• Simplify pipeline management by dynamically generating stages based on the tenants parameters.
• Enhance reusability by using a common stage template for all stages, reducing duplication.
• Allowing flexibility by allowing custom parameters to be passed to the stages.
• Better scalability to easily support new tenants, environments, regions or clusters.
• Consistency by ensuring all tenants and environments follow the same deployment process.

Having one centralized location for cluster orchestration allows us to easily scale and consolidate processes into a single framwork. The cluster orchestrator could also for example perform some static code analysis or linting before running a specific stage template, ensuring that all templates utilizing the cluster orchestrator adhere to the same pre-flight checks. This not only streamlines the CI/CD processes but also enhances the overall quality and reliability of our deployments.