Implementing Cloud Run canary deployments with Git branches and Cloud Build

Preparing your environment

1. In Cloud Shell, create environment variables to use throughout this tutorial:

   export PROJECT_ID=$(gcloud config get-value project)
   export PROJECT_NUMBER=$(gcloud projects describe $PROJECT_ID --format='value(projectNumber)')
   

2. Enable the following APIs:

   • Resource Manager
   • GKE
   • Cloud Build
   • Container Registry
   • Cloud Run

   gcloud services enable \
       cloudresourcemanager.googleapis.com \
       container.googleapis.com \
       secretmanager.googleapis.com \
       cloudbuild.googleapis.com \
       containerregistry.googleapis.com \
       run.googleapis.com
   

3. Grant the Cloud Run Admin role (roles/run.admin) to the Cloud Build service account

   gcloud projects add-iam-policy-binding $PROJECT_ID \
     --member=serviceAccount:$PROJECT_NUMBER@cloudbuild.gserviceaccount.com \
     --role=roles/run.admin
   

4. Grant the IAM Service Account User role (roles/iam.serviceAccountUser) to the Cloud Build service account for the Cloud Run runtime service account

   gcloud iam service-accounts add-iam-policy-binding \
   $PROJECT_NUMBER-compute@developer.gserviceaccount.com \
     --member=serviceAccount:$PROJECT_NUMBER@cloudbuild.gserviceaccount.com \
     --role=roles/iam.serviceAccountUser
   

Set Git values

If you haven't used Git in Cloud Shell previously, set the user.name and user.email values that you want to use:

    git config --global user.email YOUR_EMAIL_ADDRESS 
    git config --global user.name YOUR_USERNAME
    git config --global credential.helper store

• YOUR_EMAIL_ADDRESS: The email used with your GitHub account
• YOUR_USERNAME: Your GitHub user ID

If you're using MFA with GitHub, create a personal access token and use it as your password when interacting with GitHub through the command-line.

• Follow this link to create an access token.
• Leave the GitHub page open.

Store your GitHub user ID in an environment variable for easier access:

export GH_USER=YOUR_GITHUB_ID

Fork the project repository

To create your own writeable version of the lab repository, fork the sample repository into your GitHub account through the GitHub UI.

Clone the sample repository

Clone and prepare the sample repository:

git clone https://github.com/$GH_USER/software-delivery-workshop.git cloudrun-progression

cd cloudrun-progression/labs/cloudrun-progression

Connect your Git repository.

Cloud Build lets you create and manage connections to source code repositories using the Google Cloud console. You can create and manage connections using either the first generation or second generation of Cloud Build repositories. This tutorial uses second generation Cloud Build repositories.

Grant required permissions

To connect your GitHub host, grant the Cloud Build Connection Admin (roles/cloudbuild.connectionAdmin) role to your user account:

PN=$(gcloud projects describe ${PROJECT_ID} --format="value(projectNumber)")
CLOUD_BUILD_SERVICE_AGENT="service-${PN}@gcp-sa-cloudbuild.iam.gserviceaccount.com"
gcloud projects add-iam-policy-binding ${PROJECT_ID} \
 --member="serviceAccount:${CLOUD_BUILD_SERVICE_AGENT}" \
 --role="roles/secretmanager.admin"

Create the host connection

1. Configure the Cloud Build repository connection:

   gcloud alpha builds connections create github $GH_USER --region=us-central1
   

2. Click the link provided in the output and follow the onscreen instructions to complete the connection.

3. Verify the installation of your GitHub connection:

   gcloud alpha builds connections describe $GH_USER --region=us-central1
   

Link the sample repository

Using the host connection you just configured, link in the sample repository you forked:

 gcloud alpha builds repositories create cloudrun-progression \
     --remote-uri=https://github.com/$GH_USER/software-delivery-workshop.git \
     --connection=$GH_USER \
     --region=us-central1

Set repository name variable

Store the repository name for later use:

export REPO_NAME=projects/$PROJECT_ID/locations/us-central1/connections/$GH_USER/repositories/cloudrun-progression

Deploy your Cloud Run service

In this section, you build and deploy the initial production application that you use throughout this tutorial.

Deploy the service

• In Cloud Shell, build and deploy the application, including a service that requires authentication. To make a public service use the --allow-unauthenticated flag.

      gcloud builds submit --tag gcr.io/$PROJECT_ID/hello-cloudrun 
  
      gcloud run deploy hello-cloudrun \
        --image gcr.io/$PROJECT_ID/hello-cloudrun \
        --platform managed \
        --region us-central1 \
        --tag=prod -q
  

  The output looks like the following:

  Deploying container to Cloud Run service [hello-cloudrun] in project [sdw-mvp6] region [us-central1]
  ✓ Deploying new service... Done.
    ✓ Creating Revision...
    ✓ Routing traffic...
  Done.
  Service [hello-cloudrun] revision [hello-cloudrun-00001-tar] has been deployed and is serving 100 percent of traffic.
  Service URL: https://hello-cloudrun-apwaaxltma-uc.a.run.app
  The revision can be reached directly at https://prod---hello-cloudrun-apwaaxltma-uc.a.run.app
  

The output includes the service URL and a unique URL for the revision. Your values will differ slightly from what's indicated here.

Validate the deployment

1. After the deployment is complete, view the newly deployed service on the Revisions page in the Cloud Console.

2. In Cloud Shell, view the authenticated service response:

   PROD_URL=$(gcloud run services describe hello-cloudrun --platform managed --region us-central1 --format=json | jq --raw-output ".status.url")
   
   echo $PROD_URL
   
   curl -H "Authorization: Bearer $(gcloud auth print-identity-token)" $PROD_URL
   

Enabling Branch Deployments

In this section, you enable a unique URL for development branches in Git.

Setup Branch Trigger

Each branch is represented by a URL identified by the branch name. Commits to the branch trigger a deployment, and the updates are accessible at that same URL.

1. In Cloud Shell, set up the trigger:

   gcloud alpha builds triggers create github \
   --name=branchtrigger \
   --repository=$REPO_NAME \
   --branch-pattern='[^(?!.*main)].*' \
   --build-config=labs/cloudrun-progression/branch-cloudbuild.yaml \
   --region=us-central1
   

2. To review the trigger, go to the Cloud Build Triggers page in the Cloud Console.

Create changes on a branch

1. In Cloud Shell, create a new branch:

   git checkout -b new-feature-1
   

2. Open the sample application using your favorite editor or using the Cloud Shell IDE:

   edit app.py
   

3. In the sample application, modify line 24 to indicate v1.1 instead of v1.0:

   @app.route('/')
   
   def hello_world():
       return 'Hello World v1.1'
   
   

4. To return to your terminal, click Open Terminal.

Execute the branch trigger

1. In Cloud Shell, commit the change and push to the remote repository:

   git add . && git commit -m "updated" && git push origin new-feature-1
   

2. To review the build in progress, go to the Cloud Build Build history screen.

3. To review the new revision, after the build completes go to the Cloud Run Revisions page in the Cloud Console:

4. In Cloud Shell, get the unique URL for this branch:

   BRANCH_URL=$(gcloud run services describe hello-cloudrun --platform managed --region us-central1 --format=json | jq --raw-output ".status.traffic[] | select (.tag==\"new-feature-1\")|.url")
   
   echo $BRANCH_URL
   

5. Access the authenticated URL:

   curl -H "Authorization: Bearer $(gcloud auth print-identity-token)" $BRANCH_URL
   

   The updated response output looks like the following:

   Hello World v1.1
   

Automate canary testing

When code is released to production, it's common to release code to a small subset of live traffic before migrating all traffic to the new code base.

In this section, you implement a trigger that is activated when code is committed to the main branch. The trigger deploys the code to a unique canary URL and it routes 10% of all live traffic to the new revision.

1. In Cloud Shell, set up the branch trigger:

   gcloud alpha builds triggers create github \
     --name=maintrigger \
     --repository=$REPO_NAME \
     --branch-pattern=main \
     --build-config=labs/cloudrun-progression/main-cloudbuild.yaml \
     --region=us-central1
   

2. To review the new trigger, go to the Cloud Build Triggers page in the Cloud Console.

3. In Cloud Shell, merge the branch to the main line and push to the remote repository:

   git checkout main
   git merge new-feature-1
   git push origin main
   

4. To review the build in progress, go to the Cloud Build Builds page.

5. After the build is complete, to review the new revision, go to the Cloud Run Revisions page in the Cloud Console. Note that 90% of the traffic is routed to prod, 10% to canary, and 0% to the branch revisions.

Review the key lines of main-cloudbuild.yaml that implement the logic for the canary deploy.

Lines 39-45 deploy the new revision and use the tag flag to route traffic from the unique canary URL:

gcloud run deploy ${_SERVICE_NAME} \
--platform managed \
--region ${_REGION} \
--image gcr.io/${PROJECT_ID}/${_SERVICE_NAME} \
--tag=canary \
--no-traffic

Line 61 adds a static tag to the revision that notes the Git short SHA of the deployment:

gcloud beta run services update-traffic ${_SERVICE_NAME} --update-tags=sha-$SHORT_SHA=$${CANARY} --platform managed --region ${_REGION}

Line 62 updates the traffic to route 90% to production and 10% to canary:

gcloud run services update-traffic ${_SERVICE_NAME} --to-revisions=$${PROD}=90,$${CANARY}=10 --platform managed --region ${_REGION}

1. In Cloud Shell, get the unique URL for the canary revision:

   CANARY_URL=$(gcloud run services describe hello-cloudrun --platform managed --region us-central1 --format=json | jq --raw-output ".status.traffic[] | select (.tag==\"canary\")|.url")
   
   echo $CANARY_URL
   

2. Review the canary endpoint directly:

   curl -H "Authorization: Bearer $(gcloud auth print-identity-token)" $CANARY_URL
   

3. To see percentage-based responses, make a series of requests:

   LIVE_URL=$(gcloud run services describe hello-cloudrun --platform managed --region us-central1 --format=json | jq --raw-output ".status.url")
   for i in {0..20};do
     curl -H "Authorization: Bearer $(gcloud auth print-identity-token)" $LIVE_URL; echo \n
   done
   

Releasing to Production

After the canary deployment is validated with a small subset of traffic, you release the deployment to the remainder of the live traffic.

In this section, you set up a trigger that is activated when you create a tag in the repository. The trigger migrates 100% of traffic to the already deployed revision based on the commit SHA of the tag. Using the commit sha ensures the revision validated with canary traffic is the revision utilized for the remainder of production traffic.

1. In Cloud Shell, set up the tag trigger:

   gcloud alpha builds triggers create github \
     --name=tagtrigger \
     --repository=$REPO_NAME \
     --tag-pattern=. \
     --build-config=labs/cloudrun-progression/tag-cloudbuild.yaml \
     --region=us-central1
   

2. To review the new trigger, go to the Cloud Build Triggers page in the Cloud Console.

3. In Cloud Shell, create a new tag and push to the remote repository:

   git tag 1.1
   git push origin 1.1
   

4. To review the build in progress, go to the Cloud Build Build history screen in the Cloud Console.

5. After the build is complete, to review the new revision, go to the Cloud Run Revisions page in the Cloud Console. Note that the revision is updated to indicate the prod tag and it is serving 100% of live traffic.

6. In Cloud Shell, to see percentage-based responses, make a series of requests:

   LIVE_URL=$(gCloud Run services describe hello-cloudrun --platform managed --region us-central1 --format=json | jq --raw-output ".status.url")
   
   for i in {0..20};do
     curl -H "Authorization: Bearer $(gcloud auth print-identity-token)" $LIVE_URL; echo \n
   done
   

7. Review the key lines of tag-cloudbuild.yaml that implement the production deployment logic.

   Line 37 updates the canary revision adding the prod tag. The deployed revision is now tagged for both prod and canary:

   gcloud beta run services update-traffic ${_SERVICE_NAME} --update-tags=prod=$${CANARY} --platform managed --region ${_REGION}
   

   Line 39 updates the traffic for the base service URL to route 100% of traffic to the revision tagged as prod:

   gcloud run services update-traffic ${_SERVICE_NAME} --to-revisions=$${NEW_PROD}=100 --platform managed --region ${_REGION}