“It takes a minute to have a crush on someone,
an hour to like someone,
and a day to love someone...
but it takes a lifetime to forget someone.”

— Kahlil Gibran

Prologue

Suppose that you, my friend, are developing a software, how would you publish it? If it’s a client application, you may manually build it, upload the artifact, and that’s it. If it is a backend server… Think about it, you have to stop the current service and re-deploy it (or them?). Such tedious work.🥴

For long people are trying to automate the development process. The deployment, usually involving only some script execution, can be automated relatively easier. By automatically build and deploy our product, we are practicing the idea of Continuous Integration/Continuous Delivery, a.k.a. CI/CD. Generally speaking, it is the process of automatically build, test and eventually publish our product.

Recently, I got some work on a Web application, and with what I learned in school, I decided to use CI/CD to automatically deploy it on my server. It ROCKS!😍Therefore, I’m writing this article to record this excellent practice.

In this post, you’ll learn:

“Dockerize” a Spring Boot application.
Manage deployment with Kubernetes. (k3s to be specific.)
Deployment script practice.
Integration with GitHub CI/CD.
Some tips and troubleshoots.

So, good luck, have fun!🫡

All code for this post can be found at GitHub.

Dockerizing Spring Boot Application

In this part, I’ll talk about the very basic of deploying your application with Docker. And it will be the foundation for more advanced deployment with Kubernetes.

Install Docker

Well, although Docker is important in this post, it is not the point to go further into its installation. But it can be a problem now since, … some access restriction to the official Docker website. So you may use mirror when installing it.

# without mirror
curl -fsSL https://test.docker.com -o test-docker.sh && bash test-docker.sh
# with mirror
curl -fsSL https://get.docker.com -o get-docker.sh && bash get-docker.sh --mirror Aliyun

For more information on Docker, you can refer to Getting Started with Docker.

There maybe some network issues when pulling images, you can refer to Network Issues for more information.

Build with `Dockerfile`

Assume that you already have a Spring Boot application ready for deployment, the first thing is to use maven install to pack it into a .jar executable file as shown below. In case ignored, you can change the version in the pom.xml.

Then, we need a Dockerfile to create an image from our .jar. The image runs our application only, so the Dockerfile is quite simple, copy our .jar into it, and run it on start. And don’t forget to expose our port.

FROM openjdk:17-jdk-slim
COPY target/Deployment-0.0.1.jar /app.jar
EXPOSE 8088
CMD ["java", "-jar", "/app.jar"]

It is always a good practice to use a smaller base image, so here we use JDK slim from OpenJDK. Actually a JRE base image is better, but I didn’t find a suitable one.

And of course, you can add more arguments based on your use case.

1	CMD ["java", "-Duser.timezone=Asia/Shanghai", "-jar", "/app.jar", "--spring.profiles.active=dev"]

Then, we can build the Docker image with the following command.

1	sudo docker build -t deployment:0.0.1 .

Finally, you can test run it with docker run (not in background), and test it with curl in a new terminal session.

sudo docker run -p 8088:8088 deployment:0.0.1

# new session
curl localhost:8088/api/ping
pong

More Flexible `Dockerfile`

Although we have the application running with Docker, but we have to modify the Dockerfile for every version update. To tackle this problem, we can use ARG option in the Dockerfile. This ARG is required when building the image, you can also add a default value by using ARG VERSION=0.0.1.

FROM openjdk:17-jdk-slim
ARG VERSION
COPY target/Deployment-${VERSION}.jar /app.jar
EXPOSE 8088
CMD ["java", "-jar", "/app.jar"]

So now, we can build the image with the following command. The rest is the same as before.

1	docker build -t deployment:0.0.1 --build-arg VERSION=0.0.1 .

I think it is enough for our deployment purpose here. :)

Manage Deployment with Kubernetes

Now, its time to introduce our big ⭐ today — Kubernetes! It is most famous for the ability to automatically deploy containerized application with high scalability. It is powerful yet extremely complex, so many consider it impossible to master Kubernetes. (I love this meme. 🤣)

meme-1

Kubernetes is, however, too powerful for small applications, so we usually use a lightweight distribution — K3s. In the following sections, we’ll use k3s as demonstration. The ideas are the same, so I’ll just use Kubernetes when talking about concepts.

Install K3s

By default, K3s uses containered as the default container service. However, as the container layer is decoupled, we can also use Docker. To install K3s on your server, run the following command.

# official script
curl -sfL https://get.k3s.io | sh -

# with mirror
curl -sfL https://rancher-mirror.oss-cn-beijing.aliyuncs.com/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn sh -s - --docker

# set environment variable
echo 'export KUBECONFIG=/etc/rancher/k3s/k3s.yaml' >> ~/.bashrc
source ~/.bashrc

# test if K3s works
sudo k3s kubectl get node

Well, this is it. Next, we’re going to explore the basic concepts in Kubernetes.

Concepts

In Kubernetes, the main resources and their relations are shown in the following figure, and they can all be represented by a YAML file.

  
graph LR;
	A[Service]--expose-->B
	B[Deployment]--manage-->C
	C[Pod]--manage-->D
	D[Container]
	E[ConfigMap]--config-->B

Pod provides a direct control over containers. It works like, a wrapper in Kubernetes. A pod may contain multiple containers to ensure additional features. You can get all running pods with the following command. And as container is not actually part of Kubernetes, you can get them with Docker commands.

1 2	sudo kubectl get pods sudo docker ps

Deployment, as its name indicates, is important in deployment. It is the key for scalability as it can seamlessly scale pods up or down. I’ll show you how to create a deployment in the next section.

Finally, the Service expose the deployment, so that it can be accessed by the outside world. It ensures the deployment is consistent. For small scale deployment, however, I don’t think Service is that important, as Nginx could do the job. So I’m not going to talk about it now.

Create a Deployment

Let’s get serious. Now, you have the Docker image, how to make the best of K3s to automatically deploy it? Well, talk is cheap, I just show the YAML file.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment-demo  # name of the deployment
spec:
  replicas: 1  # how many instances behind the deployment
  selector:
    matchLabels:
      app: demo  # indicates what pods to manage
  template:  # template for the pods in this deployment
    metadata:
      labels:
        app: demo  # pod label
    spec:
      hostNetwork: true  # whether share network from host
      containers:
        - name: deployment
          image: deployment:0.0.1  # the image to deploy
          envFrom:  # set container environments
            - configMapRef:
                name: deployment-config
          volumeMounts:  # set volumes to mount
            - name: data
              mountPath: /data
      volumes:
        - name: data
          hostPath:
            path: /home/tonix/docker/deploy/data
            type: DirectoryOrCreate

Some thing to explain. K3s manages containers in its own way, so we have to convert Docker configurations into K3s equivalent. The most common use is -e and -v to set environment variables and mounted volumes respectively.

To set environment variables for containers, you need to use config map. Here we import environments from deployment-config, so let’s see its definition. All environments are defined in data section.

apiVersion: v1
kind: ConfigMap
metadata:
  name: deployment-config  # name of the config map
data:
  PROFILE: dev

The reason we use such an environment is to allow different active profile for our Spring Boot application.

To mount volumes, you just define it in the deployment.yaml.

One more thing, pay attention to hostNetwork. If your application need to access database or whatever needs network connection, you should set it true.

Finally, you can deploy your application by applying these two YAML files. To delete the deployment, just replace apply with delete.

sudo kubectl apply -f env.yaml
sudo kubectl apply -f deployment.yaml

sudo kubectl get deployments  # view all deployments
NAME              READY   UP-TO-DATE   AVAILABLE   AGE
deployment-demo   1/1     1            1           2m9s

And you can again test it with curl localhost:8088/api/ping.

If you updated your YAML files, there is no need to delete them before applying. K3s will automatically update it for you. However, if you updated your image (without changing the label), you have to delete the deployment first. Because K3s only update when the YAML file changes.

Deployment Script Breakdown

Now that you are familiar with every part of the deployment, let’s make a script to do it automatically.

Script Overview

In the script, we are going to cover these steps.

Automatically discover latest version and artifact.
Build Docker image.
Apply new deployment.

The first step is a little tricky, as it automatically choose the latest version of your application.

At the same time, we want to log our deployment into a file.

Version Discovery

We use regular expression to find version in our published artifact, and choose the latest version to deploy.

# iterate all files under `target' to find the latest version
files=`ls target/`
max_version=0
max_file=""
for file in $files; do
    version=`echo $file | grep -oE "[0-9]+\.[0-9]+\.[0-9]+"`
    if [ "$version" \> "$max_version" ]; then
        max_version=$version
        max_file=$file
    fi
done
if [ -z $max_file ]; then
    echo "[ERROR] No file to deploy" | tee -a deploy.log
    exit 1
fi
echo "[INFO] Deploying $max_file" | tee -a deploy.log

tee is a useful command when you want the output to be displayed on the terminal and wrote into a file.

Build Docker Image

In this step, we use the $max_version as the argument to build our docker image.

if [ ! -f Dockerfile ]; then
    echo "[ERROR] Dockerfile not found" | tee -a deploy.log
    exit 1
fi
echo "[INFO] Building docker" | tee -a deploy.log
sudo docker build -t deployment:$max_version --build-arg VERSION=$max_version .
if [ $? -ne 0 ]; then
    echo "[ERROR] Docker build failed" | tee -a deploy.log
    exit 1
fi

Update Deployment

Since we may change the version of our Docker image, so we’d better not hard code it in the YAML file. However, K3s doesn’t support argument when apply a new deployment, so we have to resolve it manually.

To address this problem, we rename our YAML file to deployment.template.yaml, and replace the image version with a safe placeholder {VERSION}. Then every time we create new deployment, we generate a new YAML file with the correct version. And since our Docker image changes on deployment, we should delete the previous deployment so that K3s can apply the latest deployment.

# constructing new deployment.yaml by replacing {VERSION} with $max_version
echo "[INFO] Constructing new deployment.yaml" | tee -a deploy.log
touch deployment.yaml
mv deployment.yaml deployment.yaml.old
cat deployment.template.yaml | sed "s/{VERSION}/$max_version/g" > deployment.yaml

# reload if deployment.yaml is different from deployment.yaml.old
if cmp -s deployment.yaml deployment.yaml.old; then
    echo "[INFO] No change in deployment.yaml" | tee -a deploy.log
    echo "[WARNING] Deleting previous deployment" | tee -a deploy.log
    echo "sudo kubectl delete -f deployment.yaml" | tee -a deploy.log
    sudo kubectl delete -f deployment.yaml
fi
rm deployment.yaml.old

At last, we can apply the new deployment.yaml.

echo "[INFO] Applying deployment.yaml" | tee -a deploy.log
echo sudo kubectl apply -f env.yaml | tee -a deploy.log
echo sudo kubectl apply -f deployment.yaml | tee -a deploy.log
sudo kubectl apply -f deployment.yaml
if [ $? -ne 0 ]; then
    echo "[ERROR] Failed to apply deployment.yaml" | tee -a deploy.log
    exit 1
fi

What? You want a complete deployment script? Well, here it is.😉

#!/bin/bash

echo "===== `date`" | tee -a deploy.log

if [ "$1" == "stop" ]; then
    echo "[WARNING] Stopping the deployment" | tee -a deploy.log
    sudo kubectl delete -f deployment.yaml
    sudo kubectl delete -f env.yaml
    exit 0
fi

# iterate all files under `target' to find the latest version
files=`ls target/`
max_version=0
max_file=""
for file in $files; do
    version=`echo $file | grep -oE "[0-9]+\.[0-9]+\.[0-9]+"`
    if [ "$version" \> "$max_version" ]; then
        max_version=$version
        max_file=$file
    fi
done
if [ -z $max_file ]; then
    echo "[ERROR] No file to deploy" | tee -a deploy.log
    exit 1
fi
echo "[INFO] Deploying $max_file" | tee -a deploy.log

# build docker image
if [ ! -f Dockerfile ]; then
    echo "[ERROR] Dockerfile not found" | tee -a deploy.log
    exit 1
fi
echo "[INFO] Building docker" | tee -a deploy.log
sudo docker build -t deployment:$max_version --build-arg VERSION=$max_version .
if [ $? -ne 0 ]; then
    echo "[ERROR] Docker build failed" | tee -a deploy.log
    exit 1
fi

# constructing new deployment.yaml by replacing {VERSION} with $max_version
echo "[INFO] Constructing new deployment.yaml" | tee -a deploy.log
touch deployment.yaml
mv deployment.yaml deployment.yaml.old
cat deployment.template.yaml | sed "s/{VERSION}/$max_version/g" > deployment.yaml

# reload if deployment.yaml is different from deployment.yaml.old
if cmp -s deployment.yaml deployment.yaml.old; then
    echo "[INFO] No change in deployment.yaml" | tee -a deploy.log
    echo "[WARNING] Deleting previous deployment" | tee -a deploy.log
    echo "sudo kubectl delete -f deployment.yaml" | tee -a deploy.log
    sudo kubectl delete -f deployment.yaml
fi
rm deployment.yaml.old

# apply new deployment.yaml
echo "[INFO] Applying deployment.yaml" | tee -a deploy.log
sudo kubectl apply -f env.yaml
if [ $? -ne 0 ]; then
    echo "[ERROR] Failed to apply env.yaml" | tee -a deploy.log
    exit 1
fi
sudo kubectl apply -f deployment.yaml
if [ $? -ne 0 ]; then
    echo "[ERROR] Failed to apply deployment.yaml" | tee -a deploy.log
    exit 1
fi

echo "[INFO] Completed deployment of $max_version" | tee -a deploy.log
echo "" | tee -a deploy.log

Now, you are able to deploy your application with a single command!🎊

GitHub CI/CD

Although you can deploy with a script, you still have to manually build the artifact, upload it to the server and execute the script. How can the whole bunch of these errands be completely automated? The answer is — CI/CD pipeline. In this section, I’ll show you how to use GitHub CI/CD to enable automatic deployment.

GitHub Workflow

The first thing is to add a GitHub workflow file to your repository to enable GitHub Action. This example shows the minimum steps to build and publish a Spring Boot application. It says build and publish our product on every push to the release branch.

name: Build and Deploy

on:
  push:
    branches:
      - release

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v2

      - name: Set up JDK 17
        uses: actions/setup-java@v2
        with:
          java-version: '17'
          distribution: 'adopt'

      - name: Build with Maven
        run: mvn clean package -DskipTests

      - name: Upload to server
        uses: appleboy/scp-action@v0.1.7
        with:
          host: ${{ secrets.HOST }}
          username: ${{ secrets.USERNAME }}
          key: ${{ secrets.SSH_PRIVATE_KEY }}
          port: 22
          source: "target/*.jar,Dockerfile,deployment.template.yaml,env.yaml"
          target: ${{ secrets.DEPLOY_PATH }}

  deploy:
    runs-on: ubuntu-latest

    needs: build

    steps:
      - name: Run deploy script
        uses: appleboy/ssh-action@v1.0.3
        with:
          host: ${{ secrets.HOST }}
          username: ${{ secrets.USERNAME }}
          key: ${{ secrets.SSH_PRIVATE_KEY }}
          port: 22
          script: |
            cd ${{ secrets.DEPLOY_PATH }}
            bash ./deploy.sh

In the step “Build with Maven”, we add -DskipTests because it makes no sense to test a program on deployment. And tests may fail if it involves database connection or environment sensitive operations. So just skip them, test before publishing.

When build is complete, we upload the artifact to the server, and run the deploy.sh we wrote in the last chapter. Here, it involves four GitHub action secrets. HOST, USERNAME, and SSH_PRIVATE_KEY are used to establish connection with the server while DEPLOY_PATH indicates where you want your application be deployed.

SSH Configuration

I think there is no need to explain HOST and USERNAME. If you want use a normal user instead of root, refer to Create Sudo-Enabled User on Linux. If you currently have a user, you may need to disable the password prompt, see Step 4. sudo without password.

Then, the most important part comes to the SSH_PRIVATE_KEY. This allows GitHub action to be authenticated to access your server.

First, generate a SSH key pair on your PC (not the server). This is because we want to simulate a user login for GitHub action. And ed25519 algorithm is recommended. (You can choose RSA anyway.)

1	ssh-keygen -t ed25519

It will generate id_ed25519 and id_ed25519.pub under ~/.ssh (Windows is %USERPROFILE%/.ssh) directory. The content of id_ed25519 is the SSH_PRIVATE_KEY. Then, we need to copy the public key to the server so that our private key will be authenticated.

1	ssh-copy-id username@host

For Windows users, there is an equivalent command combo to achieve this.

1	cat ~/.ssh/id_ed25519.pub \| ssh username@host "cat >> ~/.ssh/authorized_keys"

In Windows, the EOL maybe CRLF, which will appear as ^M in Linux. You can edit authorized_keys manually after copy.

GitHub Action Secrets

Open your repository, and add HOST, USERNAME, SSH_PRIVATE_KEY and DEPLOY_PATH to your repository secrets.

I suggest keep no new line at the end of secrets.

Trigger the Action

Everything is set, punch it! Push to the release branch and the action will run automatically.

And on our server, we can check the deploy.log.

By using .log suffix, we can get a quite colorful highlighting in Visual Studio Code.😋

Tada!🎉Now your application can be deployed automatically, and you can get rid of the distractions to your development process.😆

Troubleshoot

Network issues

Unfortunately, docker is blocked by some “unknown” forces, so you may need some mirror. I suggest you go search for them in the GitHub, or use the container acceleration service by Aliyun.

{
  "registry-mirrors": [
    "https://dockerproxy.com",
    "https://docker.m.daocloud.io"
  ]
}

After changing the mirror, restart docker service.

1 2	sudo systemctl daemon-reload sudo systemctl restart docker

Unable to access port 80

After you solved the network issue, you may soon encounter another weird problem accessing port 80. Whatever your Nginx configuration is, it will always respond with 404 page not found in plain text (not the Nginx 404 page)! Even if you use curl localhost:80 on the server. What the?🤬How is this possible?

Put it simple, k3s uses traefik by default which alters iptables to dispatch requests. It has a higher priority so requests won’t make it to Nginx. It is not a mandatory service so everything works fine when we don’t have the mirror, because k3s won’t be able to pull the image.

Details

I tried to stop Nginx, ~~though apparently not the cause,~~ but the response remains. It means there is another service online. Then I tried lsof -i:80, but found nothing. I’m afraid there’s some sort of attack on me, so I checked all running processes. Well, I did find this: traefik. What a suspicious name, traffic, huh? Hijacked my requests?

Fortunately, it is not any virus. k3s will use traefik as the default router, which alters the system iptables. And since it uses iptables, we cannot find it via lsof. It will take over port 80 and 443 so any other services listening on these two ports will no longer work.🥲

But, why we don’t have such problem before? Because of the network issue, we’re unable to pull the traefix image without a mirror. Since it is not mandatory (unlike pause), so k3s will silently ignore it, therefore everything works fine. Now, with a mirror, k3s is able to pull all these images, and as a result,💥, 404 not found.

How ironic, the solution, is also, the poison. By the way, it seems only the pause service is required,

Below are two articles that inspired me, much thanks.🙏

To disable traefik, one should follow these steps.

Stop k3s

1 2	sudo kubectl -n kube-system delete helmcharts.helm.cattle.io traefik sudo service k3s stop

Disable traefik in service configuration

Edit the k3s service at /etc/systemd/system/k3s.service, scroll to the bottom and add --disable=traefik option.

ExecStart=/usr/local/bin/k3s \
    server \
        '--docker' \
        '--disable=traefik' \

Then, reload services.

1	sudo systemctl daemon-reload

It is optional to remove the traefik configuration file.

1	sudo rm /var/lib/rancher/k3s/server/manifests/traefik.yaml

Restart k3s service

Finally, start k3s and you won’t have the traefik service.

1	sudo service k3s start

Now, you’ll again have control over port 80 and 443.

By the way, K3s enables many components which may not be required, so you can just disable them all. See the official document here.

ExecStart=/usr/local/bin/k3s \
    server \
        '--docker' \
        '--disable=traefik' \
        '--disable=local-storage' \
        '--disable=coredns' \
        '--disable=metrics-server' \
        '--disable=servicelb' \

Epilogue

I think it is one of the longest post I have? This is the most comprehensive summary of my deployment experience so far, and I hope it could be useful for me, and of course for you.😆Before, I was a little bit dismissive of Docker, but now, I am impressed by its portability and ease of use. It’s really convenient for scalable deployment.

But I am still NOT going to install Docker Desktop on my Windows PC!😠

Well, I guess this is it? See you around. ᓚᘏᗢ