How to Build and Deploy Docker Images on GKE — Part II
Introduction:
This blog post series covers steps from building a Docker image to deploying and running it using GKE. Even if you want to deploy on any other cloud, the concept will be somewhat similar.
This article is the second post in the series. If you already have a Docker image, then follow along as we cover Step 3. If you have not yet built your Docker image, refer to Part 1, where we review building a Docker image.
I have tried to keep this series less technical and less about the background on the design of GKE.
NOTE: Delete ALL GCP Clusters and containers once you have completed the exercise.
Let’s get started:
Step 1: Create a simple Python script that will be dockerized or containerized in an image. This may be a Python script or any other code; the steps will be the same. Ensure what you want to build is running successfully on your local machine and be aware of the dependencies.
If you are interested in getting started with Snowflake Data Warehouse using Python, refer to our earlier article.
Step 2: Build and verify the Docker image.
(Skip step 1 and 2 if you already have a working Docker image)
PART 2
Step 3: Set up GKE and deploy the Docker image on it.
Prerequisite: This is a fairly simple task. A basic familiarity with Docker, Kubernetes, or any cloud service provider will be sufficient.
Level: Beginner
Resources:
Github → https://github.com/architpandita/sample-app
GKE → https://cloud.google.com/kubernetes-engine/docs/tutorials/hello-app
Docker Setup → https://docs.docker.com/get-started/
Kubernetes → https://kubernetes.io/docs/tutorials/kubernetes-basics/
I highly recommended these videos:
Let’s Begin! PART 2
Step 1: Create an account
Create an account on https://console.cloud.google.com/
They provide a free tier account valid for a year with 4 nodes and $300 credit.
Basic components of GCP:
Compute: This is the place where you create a VM instance.
Billing: You can track your credits available and computation cost as well.
Kubernetes: This section is where we are going to deploy our Docker image
Step 2: Log in to your account to create a project and VM instance
Create a project
Click on “Select a project” in the top left ribbon of the window. A pop-up window will open where you will be able to create a new project by clicking “New Project” in the top right corner of the pop-up window. You may change the config if needed.
Once the project is created, ensure the project you created is selected in the ribbon.
Store Project ID
Note: Project ID has to be passed in Kubernetes, which is unique in nature globally
Go to Home under the popup menu with the three lines in the top left corner. Click on your project in the ribbon. In the details, you will find the project id.
Step 3: Create a compute instance
The compute instance is needed to connect with the local cmd and to run gcloud commands.
a) Click on the 3 lines(menu) at the top left corner to open options.
b) Click on Compute Engine.
c) Click VM instance and then create.
Select the config of the instance. I am using the default values.
You can change the zone, but the cost will also be changed accordingly.
d) Click “Create” at the bottom of the page and wait for the instance to process.
e) The instance is up and running once the green tick appears next to your instance. Click on the SSH drop-down or the “Connect” button, then select “View gcloud command.”
f) Copy the command and paste in the terminal or command prompt or within the GCP Shell, which is installed below:
Step 4: Connect via gcloud CLI
GCP has various methods to connect and execute commands either from the web terminal or by setting up gcloud SDK on the local system. This second approach is the method used here:
Installation guide: https://cloud.google.com/sdk/docs/quickstarts
E.g., Windows:
go to: https://cloud.google.com/sdk/docs/quickstart-windows
Download the SDK and install it.
After Installation
a) open Google Cloud SDK Shell and Run “ gcloud init”
(if the command “gcloud” is not recognizable, go to the folder where the shell is installed and run the shell from there)
This gcloud will setup and authorize the user, which will redirect to the window where you will authorize the application.
Multiple options are provided in this window, like which project to choose and choosing a compute zone.
Once we are logged in and authorized in cmd, continue to the next step.
b) Run cmd “ gcloud components list.”
This will give the list of all installed and uninstalled components. Since it’s your first time, the two required components, Kubernetes to deploy and Docker GCR(contains the image), are not installed.
c) Run cmd “ gcloud components install kubectl.”
d) Run cmd “ gcloud components install docker-credential-gcr.”
After a successful installation, your screen should match the image below.
e) Run cmd “ gcloud components list.”
Now we have setup Gcloud properly. Project and zone are selected via “gcloud init.”
Step 5: Push the Docker image to GCR
GKE (Google Kubernetes Engine) run image present in Google Container Registry (GCR). This is where we push our Docker image.
Check if the Docker image is presently using “ docker image ls.”
a) Run cmd: “ gcloud auth configure-docker ” to authorize
Tag the image for GCR:
“ docker tag <docker image>:<Tag> gcr.io/<project ID>/<name> ”
b) Run cmd “ docker tag sample-app:latest gcr.io/hip-bonito-273110/sample-app.”
c) Push image to GCR (this will take time-based on internet speed):
Run cmd “ docker push gcr.io/hip-bonito-273110/sample-app.”
Note: Project ID has to be passed, which is unique in nature globally.
Now we have successfully sent the docker image to GCR. From there, Kubernetes will run the image.
Step 6: Kubernetes deployment
Open the Kubernetes Engine Page from the option Menu.
Before deployment:
a) Set config parameters:
“ gcloud config set project $PROJECT_ID “
“ gcloud config set compute/zone [COMPUTE_ENGINE_ZONE] “
b) Create a cluster for Kubernetes:
run cmd: “gcloud container clusters create sample-cluster — num-nodes=2”
(This will take time to come up)
Note: We have four free nodes and limited memory. Try to limit this to two nodes since the current config allows three nodes. Using more than two nodes may cause resource scarcity.
If you receive the error “ insufficient Region Quota,” all nodes may be in use. Check the Kubernetes dashboard for anything running that may be unnecessary and proceed to delete. Try again once deleted; the dashboard will be new again.
c) Verify by cmd:
run cmd: “ gcloud compute instances list.”
or
Check the web page of the Kubernetes cluster:
d) Deploy the app as workload:
Run cmd: “ kubectl create deployment sample-app — image=gcr.io/<projectID>/sample-app:latest ”
Use “ — restart=Never ” when your application job is to perform some task and then close upon completion
Or
via web
e) Click on “select” to select the image we have pushed or the image we want to run
Provide application name and click “ DEPLOY.”
Wait for the pods to be prepared.
Key points post-preparation
In this application, we are expecting this kind of error. If you check the Python script we have in Docker, the Docker image will execute once. At which point, there are no other commands to be run. The Docker image is destroyed or closes once all the tasks are completed as it is not continuously alive in Docker. This seems like an error. However, the script has run successfully in the back end, and the work is finished.
Another thing to note is that, as you can see, Kubernetes tried to restart many times. This will cause the script/ docker image to run multiple times. We can verify this by seeing the logs of pods we have created. Click on the POD in Managed Pods shown in the above image. Notice the time and logs have stored the python scripts output multiple times.
If there is any other error, you will find it in the logs.
Notice the time and logs have stored the Python scripts output multiple times.
Delete ALL Clusters and containers once you have completed the exercise in order to conserve GCP credits.
Final Thoughts
Docker and Kubernetes have revolutionized the IT industry, enabling the adoption of DevOps techniques simultaneously with cloud technology. They have gained significant market momentum and have become a great recipe for success today. Therefore, it becomes more important to understand these technologies at a basic level as a manager, tester, developer, and/or infrastructure support team member.
In Part 1, we covered understanding Docker, wrote a Python script, and containerized it. We covered these steps so it can be run almost everywhere with minimal setup. In this article, we looked at how to scale and run to meet large user requirements using the “consumption-based, pay-as-you-go” model.
Ready to Accelerate Your Digital Transformation?
At Hashmap, we work with our clients to build better together.
If you are considering moving data and analytics products and applications to the cloud or if you would like help and guidance and a few best practices in delivering higher value outcomes in your existing cloud program, please contact us.
Hashmap, an NTT DATA Company, offers a range of enablement workshops and assessment services, cloud modernization and migration services, and consulting service packages as part of our Cloud (and Snowflake) service offerings. We would be glad to work through your specific requirements. Reach out to us here.
Other Tools and Content You Might Like
Feel free to share on other channels, and be sure and keep up with all new content from Hashmap here. To listen in on a casual conversation about all things data engineering and the cloud, check out Hashmap’s podcast Hashmap on Tap as well on Spotify, Apple, Google, and other popular streaming apps.
Archit Pandita is a Python developer and Data Engineer with Hashmap, an NTT DATA Company, providing Data, Cloud, IoT and AI/ML solutions and consulting expertise across industries with a group of innovative technologists and domain experts accelerating high-value business outcomes for our customers.
Have a question? Don’t hesitate to reach out to connect or exchange more information. I’m happy to help: Archit’s LinkedIn
