Blue Green Deployment with Github Actions

10 Mar 2021

Reading time ~2 minutes

There are many ways to do blue/green deployment. This is the way we do it; but using Github Actions, AWS route53 weighted DNS, and ECS.

The setup

Our ECS cluster qa-default-01 will have 2 identical services.

qa-rest-service-blue
qa-rest-service-green

They each have their own application load balancer.

qa-rest-service-blue   qa-rest-service-blue-***.us-west-2.elb.amazonaws.com
qa-rest-service-green  qa-rest-service-green-***.us-west-2.elb.amazonaws.com

We have an endpoint DNS record that points to both of the 2 ALB endpoints with weighted DNS setup.

rest-service.qa.my-company.com      CNAME  300  WRR=100  Public  *************  qa-rest-service-blue-***.us-west-2.elb.amazonaws.com.
rest-service.qa.my-company.com      CNAME  300  WRR=0    Public  *************  qa-rest-service-green-***.us-west-2.elb.amazonaws.com.

In this setup, we have all the traffic going to qa-rest-service-blue. We will call that the primary cluster while qa-rest-service-green will be secondary.

Github Actions

Define some variables for the ECS cluster/service.

env:
  AWS_ACCOUNT_ID_QA: 123456789012
  AWS_DEFAULT_REGION: us-west-2
  AWS_DEFAULT_OUTPUT: json

  HOSTED_ZONE_ID: Z1234567890ABC
  ZONE_NAME: rest-service.qa.my-company.com.
  ECS_CLUSTER: qa-default-01
  SERVICE_BLUE: qa-rest-service-blue
  SERVICE_GREEN: qa-rest-service-green
  REPO_NAME: rest-service

You can check out the custom github action here: cwong47/action-deploy-ecs-blue-green

Notice! I use bash because that is what I normally do, but feel free to write the script out in your favorite language.

jobs:
  deploy-blue-green:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2

      - name: Get blue-green status
        uses: "cwong47/action-deploy-ecs-blue-green@latest"
        id: get-blue-green-info
        with:
          action: get-blue-green-info
          hosted_zone_id: ${{ env.HOSTED_ZONE_ID }}
          zone_name: ${{ env.ZONE_NAME }}
          ecs_service_blue: ${{ env.SERVICE_BLUE }}
          ecs_service_green: ${{ env.SERVICE_GREEN }}
          ecs_cluster: ${{ env.ECS_CLUSTER }}

Scale up the original secondary cluster to match the min/max capacity of the primary.

      - name: Scale up original secondary cluster
        uses: "cwong47/action-deploy-ecs-blue-green@latest"
        id: update-secondary-autoscale-capacity
        with:
          action: update-autoscale-capacity
          ecs_cluster: ${{ env.ECS_CLUSTER }}
          ecs_service: ${{ steps.get-blue-green-info.outputs.original_secondary_service }}
          min_capacity: ${{ steps.get-blue-green-info.outputs.original_primary_min_capacity }}
          max_capacity: ${{ steps.get-blue-green-info.outputs.original_primary_max_capacity }}

Download secondary task definition from AWS, update image tag, upload back to AWS for ECS deployment.

        - name: Download task definition and update deployment version
          id: download-task-def
          run: |
            ...

        - name: Fill in the new image ID in the Amazon ECS task definition
          id: task-def
          uses: aws-actions/amazon-ecs-render-task-definition@v1
          with:
            ...

        - name: Deploy Amazon ECS task definition
          uses: aws-actions/amazon-ecs-deploy-task-definition@v1
          with:
            ...

Once deployment is successful, we flip the DNS and the newly deployed cluster will be taking 100% of the traffic.

      - name: Update Route53 weight for blue-green
        uses: "cwong47/action-deploy-ecs-blue-green@latest"
        id: update-primary-dns-weight
        with:
          action: update-dns-weight
          hosted_zone_id: ${{ env.HOSTED_ZONE_ID }}
          primary_route53_json: ${{ steps.get-blue-green-info.outputs.pending_primary_json }}
          secondary_route53_json: ${{ steps.get-blue-green-info.outputs.pending_secondary_json }}

We can now scale down the original primary cluster just in case we want to do a quick roll-back.

      - name: Scale down original primary cluster
        uses: "cwong47/action-deploy-ecs-blue-green@latest"
        id: update-primary-autoscale-capacity
        with:
          action: update-autoscale-capacity
          ecs_cluster: ${{ env.ECS_CLUSTER }}
          ecs_service: ${{ steps.get-blue-green-info.outputs.original_primary_service }}
          min_capacity: ${{ steps.get-blue-green-info.outputs.original_secondary_min_capacity }}
          max_capacity: ${{ steps.get-blue-green-info.outputs.original_secondary_max_capacity }}

Complete YAML File

The complete jobs and actions can be found here.

github-actions-build-deploy-blue-green.yml

It includes some popular actions from the marketplace.