1. Overview
Kubernetes has become a go-to platform for managing containerized applications in the cloud. But with great power comes great responsibility: we need to keep a close eye on how our applications are performing. That’s where Prometheus comes in handy.
On the other hand, Prometheus is an open-source monitoring toolkit. It continuously tracks various metrics, providing valuable insights into how our apps are using resources like CPU and memory.
In this tutorial, we learn how to use Prometheus to monitor these critical resources within our Kubernetes pods.
2. Setting up Prometheus in Kubernetes
One of the easiest ways to get Prometheus up and running in our Kubernetes cluster is by using Helm. This is a package manager for Kubernetes, similar to how we use apt or yum on a Linux system.
First, we need to install Helm on our machine. Then, we can add the Prometheus chart repository:
$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
Next, let’s update our Helm repositories to fetch the latest charts:
$ helm repo update
Additionally, since we want to use Prometheus to monitor our Kubernetes cluster, let’s create a configuration file that Prometheus will use instead of the default setting.
We’ll name the file values.yaml, and this file will enable Prometheus to scrape Kubernetes metrics:
$ cat values.yaml
server:
extraScrapeConfigs: |
- job_name: 'kubernetes-pods'
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_label_app]
action: keep
regex: .+
Finally, let’s install Prometheus using our custom values.yaml file:
$ helm install prometheus -f values.yaml prometheus-community/prometheus
Therefore, this command installs Prometheus with the specified configuration, enabling it to scrape metrics from all pods in the cluster.
3. Understanding Prometheus Metrics for CPU and Memory
When it comes to monitoring our Kubernetes pods, understanding the relevant metrics is crucial. Two key players in this game are the container_cpu_usage_seconds_total and container_memory_working_set_bytes metrics.
Furthermore, the container_cpu_usage_seconds_total metric keeps track of the total CPU time used by a container, measured in seconds. By looking at this over time, we can see how much processing power our container is using and spot any unusual spikes or patterns.
Similarly, the container_memory_working_set_bytes metric tells us how much memory our container is actively using. This includes both the memory it needs to keep running and the memory it’s using for temporary storage (cache).
Moreover, keeping an eye on this metric helps us understand our container’s memory footprint and whether it’s staying within its limits.
Speaking of limits, the kube_pod_container_resource_limits metric is our go-to source for understanding the constraints placed on our containers. This metric tells us the maximum amount of CPU and memory each container is allowed to use.
4. Running a Basic Web Application With Prometheus Monitoring in Kubernetes
Before we consider Prometheus Query Language (PromQL), let’s set up a simple Kubernetes pod and configure Prometheus to gather the necessary metrics. This pod runs a basic NGINX web server, giving us a real-world scenario for understanding resource usage.
First, let’s create a YAML configuration file, named web-app.yaml, for our web-app pod:
$ cat web-app.yaml
apiVersion: v1
kind: Pod
metadata:
name: web-app
labels:
app: web-app
spec:
containers:
- name: web-app
image: nginx:latest
ports:
- containerPort: 80
resources:
limits:
cpu: "1"
memory: "512Mi"
requests:
cpu: "0.5"
memory: "256Mi"
This configuration defines a single-container pod running the NGINX web server, along with resource requests and limits for CPU (0.5 cores request, 1 core limit) and memory (256 MiB request, 512 MiB limit) to manage resource allocation.
Additionally, we’ve explicitly exposed port 80 in the container, which is the default port for NGINX.
To get this pod up and running, let’s apply the configuration file to our Kubernetes cluster:
$ kubectl apply -f web-app.yaml
pod/web-app created
We can confirm the pod is running:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
web-app 1/1 Running 0 75s
However, to access this pod from outside the cluster, we need to create a service to expose it:
$ kubectl expose pod web-app --type=NodePort --name web-app-service --port=80
Hence, this command creates a service, named web-app-service, that exposes the web-app pod on a port we can access from outside the cluster.
Next, let’s get the URL for accessing our web app:
$ minikube service web-app-service
|-----------|-----------------|-------------|---------------------------|
| NAMESPACE | NAME | TARGET PORT | URL |
|-----------|-----------------|-------------|---------------------------|
| default | web-app-service | 80 | http://192.168.49.2:30137 |
|-----------|-----------------|-------------|---------------------------|
🎉 Opening service default/web-app-service in default browser...
This automatically opens the NGINX welcome page on our default browser:
Now, let’s set up Prometheus to monitor our web-app pod.
We’ve already installed Prometheus, so we can access its web interface by port-forwarding:
$ kubectl port-forward svc/prometheus-server 9090:80
After running this command, we can open our browser and go to http://localhost:9090:
Therefore, with our web-app pod running and Prometheus configured, we’re ready to start writing queries to monitor the pod’s CPU and memory usage.
5. Writing Prometheus Queries for CPU Usage
Now, let’s learn how to write Prometheus queries to monitor the CPU usage of our web-app pod.
5.1. Basic CPU Usage Query
To understand how much processing power our web-app pod is using, we can query the container_cpu_usage_seconds_total metric in Prometheus. This metric keeps track of the total CPU time (in seconds) consumed by the pod’s container.
However, since container_cpu_usage_seconds_total is a cumulative metric (meaning it always increases), we need to use the rate() function to get a more meaningful measurement.
Moreover, this function calculates the per-second rate of change over a specific time interval.
First, let’s increase the traffic to the web-app pod by refreshing the NGINX welcome page repeatedly or opening the URL in multiple tabs within our browser. This generates more traffic and increases the CPU load slightly.
Next, let’s execute a simple query on our Prometheus web interface to get the average CPU usage per second over the last 5 minutes for our web-app pod:
rate(container_cpu_usage_seconds_total{pod="web-app"}[5m])
The query gives us a value of 0.0001825493896777227, representing the average number of CPU cores the pod used during that time.
Let’s see the graphical representation:
Next, let’s get even better insights by calculating the CPU usage as a percentage of the pod’s allocated resources.
5.2. CPU Usage Percentage
To get a better understanding of CPU usage relative to allocated resources, we can calculate the CPU usage percentage for the web-app pod:
100 * (sum(rate(container_cpu_usage_seconds_total{pod="web-app"}[5m])) by (pod) / sum(kube_pod_container_resource_limits{pod="web-app", resource="cpu"}) by (pod))
Hence, this query divides the average CPU usage rate by the CPU limit defined for the pod, expressing the result as a percentage. We get a value of 0.003232928830329154%.
Let’s also see the graphical representation:
This percentage offers insight into how much of the pod’s allocated CPU is being used.
6. Writing Prometheus Queries for Memory Usage
Let’s now turn our attention to monitoring the memory usage of our web-app pod.
6.1. Basic Memory Usage Query
Similarly, we can use Prometheus to query metrics related to memory consumption. The container_memory_working_set_bytes metric provides insight into the amount of memory a container is actively using.
Now, let’s get the current memory usage of our web-app pod using a simple query:
sum(container_memory_working_set_bytes{pod="web-app"})
Therefore, our current memory usage is 12492800 bytes. This query sums up the memory usage across all containers within the web-app pod, giving us the total memory being used at a specific moment.
6.2. Memory Usage Percentage
To calculate the memory usage percentage for the web-app pod, let’s leverage two key metrics: container_memory_working_set_bytes and kube_pod_container_resource_limits. The first metric provides the current memory usage in bytes, while the second gives us the memory limits configured for the pod.
Now, let’s calculate the memory usage percentage for our web-app pod:
100 * (sum(container_memory_working_set_bytes{pod="web-app"}) by (pod) / sum(kube_pod_container_resource_limits{resource="memory", pod="web-app"}) by (pod))
Consequently, this query enables us to get the memory usage percentage for the web-app pod, which is 2.300262451171875% in this case.
7. Conclusion
In this article, we learned how to use Prometheus to monitor the CPU and memory usage of Kubernetes pods. Leveraging these insights, we can optimize our Kubernetes deployments and ensure the smooth operation of our containerized applications.