Google-Cloud on Home

Cloud Native Tutorial

Tue, 07 Mar 2023 00:00:00 +0000

Introduction

Last year I created a tutorial which explains the basics of application deployment in a cloud-native environment. I have also recorded videos for the individual chapters for a Dell internal learning platform, these are not publicly available.
With this post I describe the individual chapters of the tutorial, the individual parts of the tutorial follow on from each other. The corresponding code can be found on GitHub. A simple website serves as a sample application and can be run as a container in various cloud environments. The sample application’s source code is part of the tutorial.
For simplicity, do not attempt this while using a corporate firewall. Comments, additions, and collaboration are welcome.

Find the code on GitHub

1. Setup of local environment

The tutorial provides a virtual machine (VM) to ensure a consistent development environment. This VM is deployed using Vagrant. Vagrant allows it to leverage a declarative configuration file that describes the required software, packages, and operating system configuration. The VM used in this tutorial is based on Ubuntu 20.04 LTS. The VM is allocated 1 vCPU and 4 GB of RAM.
We recommend Virtualbox as the virtualization software for this tutorial.
This tutorial uses Visual Studio Code as a code editor.

After installing the three components (Vagrant, VirtualBox, and Visual Studio Code), the GitHub repository can be cloned, and the VM can be started. The first line ensures that that the software tracking tool git is still installed if it is not already present on the PC:

winget install --id Git.Git -e --source winget
git config --global core.autocrlf false
git clone https://github.com/smichard/cloud_bites_tutorial
cd cloud_bites_tutorial
vagrant up

After the VM has started and all software components have been fully installed, you can log into the VM via ssh:

vagrant ssh

The VM has been configured to mount the host PC’s file system, it is available in the vagrant folder:

cd /vagrant

2. Build and run Docker containers locally

In this section, a container with the demo application is created first. This container is started locally can be reached via the host PC’s web browser.

Switch to the directory:

cd cloud_bites_tutorial/1_2_app_sources

View the container images available in the VM:

docker images

Build the container image based on the folder’s Dockerfile:

docker build -t <image_name> .
docker images

Running the Docker container. With the following command the container runs in the background and the VM ports 8082 are mapped to the container port 80. The website is available via the Host PC’s web browser at localhost:8082:

docker run -d -p 8082:80 <image_name>

The sample application’s source code can be modified easily with the help of a small helper script. This helper script is run twice, and the container image is rebuilt twice. The commands shown start new containers with the newly generated container images. The website is then available via the host PC’s web browser under localhost:8083 and localhost:8084:

./update_script.sh
docker build -t <image_name_2> .
docker run -d -p 8083:80 <image_name_2>
./update_script.sh
docker build -t <image_name_3> .
docker run -d -p 8084:80 <image_name_3>

Running containers can be listed and stopped with the following commands:

docker ps
docker kill <container_id>

The created container images can be pushed to the container registry if you have an account on Docker Hub. The container image must contain the Dockerhub username. So, the container image may need to be rebuilt or named appropriately via a tag:

docker login
docker push <dockerhub_username>/<image_name>

3. Deploy a local Kubernetes cluster using K3D

In the following section you are going to use the K3D project to locally deploy a minimal Kubernetes cluster. K3D is a lightweight wrapper to run K3S, Rancher Lab’s minimal Kubernetes distribution, in Docker. K3D makes it very easy to create single- and multi-node K3S clusters in Docker, e.g. for local development on Kubernetes. A good introduction to K3D with some use cases can be found in the DevOps Toolkit YouTube video.

Deploy a local K3D cluster with three control planes and three worker nodes:

k3d cluster create local-cluster --servers 3 --agents 3 -p "8080:80@loadbalancer"

The cluster and the individual nodes can be displayed with the following commands:

k3d cluster list
kubectl get nodes

The following command deploys the demo application to the local Kubernetes cluster that has just been created. This step creates a deployment with multiple pods, a service, and an ingress controller. You can find the declarative configuration in the 1_3_local_deployment/deployment.yml file:

kubectl apply -f 1_3_local_deployment/deployment.yml

The following command displays the deployment, the replica set, the pods, and the service:

kubectl get deployments,replicasets,pods,services

The website is then available via the host PC’s web browser under localhost:8080.

4. Basic operations to handle Pods and Deployments

The following section details a few basic commands for handling pods and deployments. The following commands create two pods named my-pod-1 and my-pod-2 each with the container image nginx:alpine. The last command shows all pods running in the default namespace:

kubectl run my-pod-1 --image=nginx:alpine
kubectl run my-pod-2 --image=nginx:alpine
kubectl get pods

The following command will stop and delete the pod named my-pod-1. Since this pod is not part of a ReplicaSet, it will not be restarted:

kubectl delete pod my-pod-1
kubectl get pods

The following command scales the deployment created in the previous section to six replicas. The second command displays the deployment, the replica sets, and the pods in the default namespace. Now, six pods should be shown for dpl-demo-1:

kubectl scale deployment dpl-demo-1 --replicas=6
kubectl get deployments,replicasets,pods

Now it’s time to change the deployment. The 1_3_local_deployment/deployment.yml file must be modified to do this. For example, in line 24, the container image can be changed from demo-app:sydney to demo-app:london. The adjusted deployment is rolled out again with the following command:

kubectl apply -f 1_3_local_deployment/deployment.yml
kubectl get deployments,replicasets,pods

The second command displayed the deployment, the replica sets, and the pods for the default namespace. By launching the changed deployment, a new replica set was created in which the number of pods is scaled up to the required number of pods. The number of pods for the existing replica set is scaled to zero.
The deployment is changed a second time and rolled out again. How does this occur? The container image is modified from demo-app:london to demo-app:newyork in line 24 of the 1_3_local_deployment/deployment.yml file. The adjusted deployment is rolled out again with the following command:

kubectl apply -f 1_3_local_deployment/deployment.yml
kubectl get deployments,replicasets,pods

The demo application’s website is always available via the host PC’s web browser via localhost:8080.
The following command shows a list of revisions for the dpl-demo-1 deployment. The second command displays details for a specific revision:

kubectl rollout history deployment dpl-demo-1
kubectl rollout history deployment dpl-demo-1 --revision=2

Use the following command to roll back the deployment to a specific revision:

kubectl rollout undo deployment dpl-demo-1 --to-revision=2

If the deployment is rolled back, the number of pods for the corresponding ReplicaSet is scaled, meaning no new ReplicaSet is created, but an existing one is used.

5. Deploy a remote Kubernetes cluster using Google Cloud – part 1

In the following section, a Kubernetes cluster is deployed in a public cloud. For this purpose, we use Google Cloud in this tutorial. For simplicity, we assume that a Google Cloud account already exists.

Warning

Costs are generated at the public cloud provider used in the following tutorial sections. It is, therefore, vital to limit the runtime of the clusters and prevent possible idle time to minimize costs.

First, obtain access to your Google Cloud account:

gcloud auth login

The following command creates a new Google Cloud project, generating a random project name. Last, the newly created project is specified in the active configuration.

PROJECT_ID="gcp-$(($(date +%s%d)/1000000))$(($RANDOM%20))" && echo $PROJECT_ID
gcloud projects create ${PROJECT_ID} --name "${PROJECT_ID}"
gcloud config set project ${PROJECT_ID}

Before proceeding with the tutorial, make sure to enable billing on the new project.

Then some required Google Cloud APIs have to be activated:

gcloud services enable compute.googleapis.com container.googleapis.com

For simplicity let’s define a default compute region and a default compute zone:

gcloud config set compute/region europe-west3
gcloud config set compute/zone europe-west3a

Finally, the Kubernetes cluster can be deployed:

gcloud beta container clusters create my-gke-cluster --zone europe-west3-a

To interact with the newly created Kubernetes Cluster, fetch its credentials. This commands updates the local kubeconfig file with the appropriate credentials and endpoint information:

gcloud container clusters get-credentials my-gke-cluster

Now the demo application can be deployed on the newly created cluster:

kubectl apply -f 1_5_cloud_deployment/deployment.yml

A load balancer was created as a networking service in the previous step. After a certain wait time, a public IP address is displayed. The website of the demo application can then be reached via a web browser:

kubectl get services

The Kubernetes cluster can be deleted with the following command:

gcloud container clusters delete my-gke-cluster

6. Deploy a remote Kubernetes cluster using Google Cloud – part 2

In this section, another Kubernetes cluster is created on Google Cloud. This time, the Cloud Console is used:

Google Cloud Screenshot

Once the Kubernetes Cluster has been created successfully, fetch its credentials:

gcloud container clusters get-credentials my-gke-cluster-2

Now the demo application can be deployed on the newly created cluster:

kubectl apply -f 1_5_cloud_deployment/deployment.yml

kubectl get services

Tip

It is crucial to delete the Kubernetes cluster if unused to keep costs low.

7. Deploy a remote Kubernetes cluster leveraging Terraform

In this section a Kubernetes cluster is created on Google Cloud leveraging Terraform. Terraform is an infrastructure as code tool allowing one to build infrastructure safely and efficiently in a declarative way on various platforms.
First the Google Cloud environment has to be prepared by enabling a couple of API’s:

gcloud services enable compute.googleapis.com container.googleapis.com

The following command authorizes the Google Cloud SDK to access Google Cloud using your user account credentials. This step adds your account to the Application Default Credentials, allowing Terraform to access these credentials to provision resources on Google Cloud:

gcloud auth application-default login

The following command initializes a working directory containing Terraform configuration files:

cd 1_7_terraform_deployment
terraform init

The terraform plan command creates an execution plan, which lets you preview the changes that Terraform plans to make to your infrastructure. By default, when Terraform creates a plan it:

Reads the current state of any already-existing remote objects to make sure that the Terraform state is up-to-date.
Compares the current configuration to the prior state and noting any differences.
Proposes a set of change actions that should, if applied, make the remote objects match the configuration. With the following command, you will create a Terraform plan. You must pass the project ID generated earlier as a variable:

terraform plan -var project_id=${PROJECT_ID}

The terraform apply command executes the actions proposed in a Terraform plan.

terraform apply -var project_id=${PROJECT_ID}

After creating the Kubernetes Cluster successfully, you can fetch its credentials:

gcloud container clusters get-credentials my-terraform-cluster --region europe-west3

Now the demo application can be deployed on the newly created cluster:

kubectl apply -f 1_5_cloud_deployment/deployment.yml

kubectl get services

The terraform destroy command conveniently eliminates all remote objects managed by a particular Terraform configuration:

terraform destroy -var project_id=${PROJECT_ID}

If you don’t need your Google Cloud project anymore, you can delete the project:

gcloud projects delete ${PROJECT_ID}

8. Visualize Kubernetes workloads with VMware Octant

You will explore, in this section, the various Kubernetes clusters created earlier with VMware Octant. Octant is a tool for developers to understand how applications run on a Kubernetes cluster. It aims to be part of the developer’s toolkit for gaining insight and approaching complexity found in Kubernetes. Octant offers a combination of introspective tooling, cluster navigation, and object management. It also provides a plugin system to extend its capabilities further.

Start Octant:

octant

The Octant application can be reached via a web browser of the host PC using localhost:8001.

Summary

The tutorial gives a quick overview of the basics of application deployment in a cloud-native environment. This tutorial is intended to provide an easy introduction with a quick learning curve. There is by far no claim to completeness. There are certainly ways to improve or expand the tutorial.
You can download the slides I used to present the tutorial via the following link:

Download slide deck

References

GitHub - link
Cloud-Native Computing Foundation - link
Virtualbox - link
Vagrant - link
K3D - link
Terraform - link
Visual Studio Code - link
VMware Octant - link
Introduction to K3D on Youtube - link
Introduction Terraform on Youtube - link

Website setup and workflow

Fri, 06 Jan 2023 00:00:00 +0000

Introduction

In this first blog post, I want to describe the workflow and the tools I use to operate this website and how I publish new content. My focus with this workflow is that I want to be able to publish new posts very quickly. Furthermore, the maintenance should be done effortlessly and I want to minimize the dependence on a hosting provider. Last but not least, the website should be very cost-efficient to run.

Tools

Being a big fan of Google Cloud Platform, I wanted to use Google Cloud Build and Google Cloud Run. Overall, I use the following tools:

Hugo - as static site generator
GitHub - as a version control solution
Google Cloud Build - to automatically build container images and deploy containers to Google Cloud Run
Google Cloud Container Registry - to store the generated container images
Google Cloud Run - to run the container

How it’s all connected

In the following section, I am going to describe how the tools work together to enable the workflow to publish the website easily and automatically. The source code for a sample website is hosted on GitHub - link.

Hugo basics

A comprehensive introduction to the Hugo framework is beyond the scope of this post. Therefore, only the essential aspects will be presented here.
Once Hugo is installed on the PC, a new website can be created with the following command. This command creates the required structure and all essential files:

hugo new site <site_name>

With the following command, the web page is generated and can be viewed in a browser. By default, a web server is created such that the website is available at localhost:1313. The view is live, which means that changes to the configuration files or the text contributions are immediately visible:

hugo server

New posts can be created using the following command, this command generates a new file in the posts folder:

hugo new posts/<post_name>.md

Themes are stored in the themes folder. Typically these are added as git submodules, this has the advantage that changes to the original theme can be included via a git merge command:

git submodule add <theme_link> themes/<theme_name>

If the web page is to be published, the following command can be executed. In this step, all static web pages are generated and stored in the public folder. The contents of this folder will eventually be published:

hugo

Leveraging container technology

To ensure the portability of the website, a container image is generated in the workflow. The following Dockerfile is used for this purpose:

FROM klakegg/hugo:0.111.3-ext-alpine-onbuild AS hugo

FROM quay.io/michard/nginx_base_image:0.2.8
COPY --from=hugo /target /usr/share/nginx/html

In this case, a Docker multi-stage build is used. Multi-stage builds allow each stage to produce a set of intermediate image layers that can be used as the base for the next stage. This allows the creation of a final image that only includes the necessary dependencies and components resulting in a smaller, more lightweight image.
The first line takes care of building the static web pages, as described at the end of the previous section by the hugo command. In the following, the result of the build process is copied to the /usr/share/nginx/html folder of an nginx based container image.
The choice of Alpine Linux based container images leads to very lightweight and secure container images. On the one hand, this ensures short build times and, on the other hand, that small capacity is occupied in the container registry.

The following commands can be used to create the container image locally and run the container locally. The website should then be available on localhost:8080:

docker build -t <image_name> .
docker run -d -p 8080:8080 <image_name>

Leveraging Google Cloud Builds and Google Cloud Run

The last section described why container technology is used and how the final container image can be generated locally. In this section, it is described how the final image can be generated leveraging Google Cloud Build. To use Google Cloud Build a cloudbuild.yaml configuration file is required. This file defines several steps of the total build pipeline, the individual steps are executed one after the other. Using a cloudbuild.yaml file allows the definition of the complete build process in a declarative way, making it easy to repeat and automate the build and deployment process.
The content of the cloudbuild.yaml is as follows:

YAML cloudbuild.yaml

steps:
# This step fetches git submodules
- name: gcr.io/cloud-builders/git
 id: Git
 args:
 - 'submodule'
 - 'update'
 - '--init'
 - '--recursive'

# This step builds the container image
- name: 'gcr.io/cloud-builders/docker'
 id: Build
 args:
 - 'build'
 - '-t'
 - 'eu.gcr.io/${PROJECT_ID}/hugo-website:${BUILD_ID}'
 - '.'

# This step pushes the image to Container Registry
# The PROJECT_ID and BUILD_ID variables are automatically
# replaced by Cloud Build.
- name: 'gcr.io/cloud-builders/docker'
 id: Push
 args:
 - 'push'
 - 'eu.gcr.io/${PROJECT_ID}/hugo-website:${BUILD_ID}'

# deploy container image to Google Cloud Run
- name: 'gcr.io/cloud-builders/gcloud'
 id: Deploy
 args:
 - 'run'
 - 'deploy'
 - 'hugo-website'
 - '--image'
 - 'eu.gcr.io/${PROJECT_ID}/hugo-website:${BUILD_ID}'
 - '--region'
 - 'europe-north1'
 - '--platform'
 - 'managed'
 - '--allow-unauthenticated'

The individual steps are described below:

the first step ensures that git submodules are also pulled. This step is necessary because the themes are usually included as git submodules, as described earlier.
the second step creates the final container image using the Dockerfile previously described. The name of the container image is composed of several elements. At the beginning, the region of the Google Cloud Container Registry is used, followed by the Project ID of the Google Cloud Project. This is followed by a freely selectable name, in this case hugo-website, followed by the current build ID, which is used as a tag.
the third step pushes the created container image into the Google Cloud Container Registry.
the fourth step deploys the created container image to Google Cloud Run. The container is launched in a managed Kubernetes cluster. In this step the container image is referenced, a name is used which in this case is hugo-website, and finally, the region in which the container is to be started is specified since this is to be a publicly accessible website, it is defined that users do not have to authenticate themselves.

In the last step, Google Cloud Run automatically generates a URL where the web page can be accessed. Using the Manage Custom Domain function and then Add Mapping in the Google Cloud Console, a custom domain can be linked to the Google Cloud Run service. To do this, the domain must be verified once with Google Cloud. To use a specific subdomain, a CNAME entry must then be set to ghs.googlehosted.com at the domain provider.

Mapping a custom domain to the Google Cloud Run service

Connecting GitHub and Google Cloud Builds

This section focuses on the missing element that completes the workflow by connecting GitHub with Google Cloud Build.
To do this, a trigger must be set up in the Google Cloud Console at the Google Build Service. A name must be selected for this trigger, then it must be defined which event should activate this trigger. For the given case, the option Push to branch is selected here. Under Source, the GitHub repository and the branch can be selected. Finally, the reference to the cloudbuild.yaml file must be created.

Creating a trigger to automatically deploy the website on each git commit

This completes the workflow. Now with each git commit towards the GitHub repository, the Google Cloud Build process is triggered, which creates the container image and then deploys the container to Google Cloud Run.

Summary

Presentation of the complete workflow

In this post, I describe the workflow I use to publish this website. For me, it is the ideal workflow. I like the high degree of automation which allows me to publish fast and effortlessly. Maintaining this workflow requires negligible effort and running the workflow and the website is very cost-efficient.

References

Hugo Documentation - link
Demo Code - link
Google Cloud Build documentation - link
Google Container Registry documentation - link
Google Cloud Run documentation - link
Docker multi-stage builds documentation - link
Git submodules documentation - link