Kubernetes HPA based on events in Google Calendar

https://kubernetes.io/

By default, HorizontalPodAutoscaler automatically scales Kubernetes Pods under ReplicationController, Deployment, or ReplicaSet controllers based on its CPU, memory, or other metrics. But if everything were so simple, I surely wouldn't write this article :)

As you probably guessed, we’ve d̶e̶c̶i̶d̶e̶d̶ needed to scale our application automatically based on events in Google Calendar.

Long story short, in our situation, it’s just impossible to scale the server-side front end by using default HPA metrics provided by Kubernetes, and I’ll tell you why.

The situation has evolved in that our application receives a large number of users only on specific days of the month when some events occur, while on other days the load is normal. Usually, in such cases, it doesn’t make any difference, we can scale applications based on default metrics such as CPU or memory while load growing, increasing the number of replicas to keep everything under control.

Why we can’t do that?

The issue lies in the peak load. It arrives very quickly, and the containers crash before scaling. Then, replicas are scaled up, and the entire load is transferred to them because previous containers are down. This cycle repeats every time. Such issues need to be fixed by changing the architecture of the application rather than relying on automation with Google Calendar. But in our situation, we didn’t have any choice and decided to automate the process of scaling using a combination of Python script and Kubernetes cron job.

Python script

The first thing we need to do in our script is to configure integration with the Google Calendar API. Using Google Cloud console we can create service account and generate JSON file for authentification.

Then, define the start and end time for the events, fetch them, and print the events as an output. Simple as that.

The full code you can find in my GitHub repo.

Integration with Kubernetes cron job

Okay, our script is up and running perfectly, the next step is to integrate it with Kubernetes and run it inside our cluster automatically. And in case we have some events for today, scale up our application replicas. Firstly, let’s Dockerize it to run in Kubernetes.

The simple Dockerfile will install all the necessary Python dependencies and execute a script when the container starts.

FROM python:3.9-slim-buster

# Set the working directory to /app
WORKDIR /app

# Copy the current directory contents into the container at /app
COPY . /app

# Install kubectl
RUN apt-get update && apt-get install -y curl && \
    curl -LO "https://dl.k8s.io/release/$(curl -Ls https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" && \
    install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl && \
    rm kubectl

RUN pip3 install google-auth google-auth-oauthlib google-auth-httplib2 google-api-python-client

# Run google-calendar.py when the container launches
CMD ["python", "google-calendar.py"]

Build a Dockerfile locally and push it to your registry.

docker build . -t <name of image>:<image tag>

docker push registry.example.com/<name of image>:<image tag>

Now we have a docker image containing our script along with all essential dependencies required for execution.

The next step is actual Kubernetes integration. For running this script inside our cluster at a specific time we’ll use a built-in resource called a cron job.

In the provided YAML file you can see a lot of interesting things, but let’s focus on highlighting the most significant ones.

schedule: Sets the schedule for the job using a cron expression. In this case, it’s scheduled to run every hour.

name, image: Names the container and specifies the Dockerized Python script image to be used.

command: Specifies the command to be executed within the container. We run a shell command that checks the output of the Python script.

serviceAccountName: Assigns the specified service account. It should have all the necessary permission to run a cron job in the Kubernetes cluster.

As you probably understand, the output is ‘No events found’ means that we don’t have any events in the Google Calendar, and our Kubernetes cron job won’t do anything, the number of replicas will remain the same. Otherwise, we have some events for today, the output will be ‘Scaling frontend replicas’, and Kubernetes will patch an application HPA by usingkubectl patch hpa command and increase the number of replicas to 10.

In the real scenario, it looks like this:

vladyslavtkachuk$ kubectl logs frontend-cron-job -n <namespace>
Scaling frontend replicas..
horizontalpodautoscaler.autoscaling/<your_app_name> patched

And number of replicas will slightly increase to 10. The logical question arising from this is: “Event day ended, what’s next”? We don’t need to run 10 replicas when the event ends, so we configured another cron job with the same functionality but scaling down.

Conclusion

In this article, we explored a unique challenge where traditional Kubernetes HPA metrics proved insufficient for dynamically scaling a server-side front end. However, it’s important to recognize that this approach is a temporary workaround rather than a fundamental solution. Despite this, the presented solution offers a practical means of automating the scaling process based on external event triggers, like Google Calendar, showing us the flexibility and extensibility of Kubernetes in real-world scenarios.