1. Introduction
In Kubernetes, a Persistent Volume Claim (PVC) is used to request storage resources from the cluster. When a PVC is created, it’s bound to a Persistent Volume (PV) that provides the actual storage. As applications run and data is written to the PVC, the available space on the underlying PV decreases. It’s important to monitor the remaining space in a PVC to ensure that applications don’t run out of storage and cause disruptions.
In this tutorial, we’ll explore different methods to find the space left in a PVC in Kubernetes.
2. Using kubectl
kubectl is a command-line tool for interacting with Kubernetes clusters. It’s the primary way for managing Kubernetes clusters, including deploying applications, inspecting and managing cluster resources, debugging, and troubleshooting.
2.1. Getting a List of PVCs
First, we get a list of all the PVCs in the cluster or a specific namespace:
$ kubectl get pvc --all-namespaces
NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES AGE
default data-pvc Bound data 5Gi RWO 12d
kube-system logs-pvc Bound logs 10Gi RWO 8d
This command displays a list of all PVCs in the cluster, along with their names, status, volume, capacity, access modes, and age.
2.2. Describe a Specific PVC
Further, we use the describe option to get detailed information about a specific PVC by name:
$ kubectl describe pvc data-pvc -n default
Name: data-pvc
Namespace: default
StorageClass: standard
Status: Bound
Volume: data
Labels: <none>
Annotations: <none>
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 5Gi
Access Modes: RWO
VolumeMode: Filesystem
Mounted By: <none>
Events: <none>
In this case, we employ the kubectl describe pvc command, the Persistent Volume Claim data-pvc name, and the namespace default to retrieve detailed information about the PVC.
Now, let’s inspect the Pods using the PVC and check their filesystems.
Thus, we list the Pods that use the PVC:
$ kubectl get pods -n default --selector=pvc=data-pvc
NAME READY STATUS RESTARTS AGE
data-pod-1 1/1 Running 0 10d
data-pod-2 1/1 Running 0 5d
As we can see, the output lists two Pods, data-pod-1 and data-pod-2.
2.3. Check Utilized Space via an exec Shell
At this point, we can select one of these Pods and enter it via kubectl exec and the respective shell path:
$ kubectl exec -it data-pod-1 -n default -- /bin/bash
Once connected, we use the du command to check the utilization of the PVC’s filesystem, depending on the path name:
$ du -sh /mnt/data
3.0Gi /mnt/data
Then, we repeat the same steps for all other Pods to get their utilization, and finally, we subtract the total utilized space from the total capacity of the PVC to find the space left.
2.4. Check Utilized Space via exec and df
Alternatively, we can execute a command within the Pod’s container and check the available disk space.
Notably, this method doesn’t work if the container doesn’t have the df command or the Pod cannot start due to an error.
For instance, we can execute the tool for data-pod-1:
$ kubectl -n default exec data-pod-1 -- df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 5Gi 3Gi 2Gi 60% /mnt/data
Let’s break this command down:
- -n specifies the namespace where the Pod is located
- exec executes a command inside a running container
- df -h is the command that runs inside the Pod
In this instance, Kubernetes executes the df -h command inside the data-pod-1 Pod in the default namespace. Thus, we get information about the storage space usage in the file systems associated with the persistent mounted volume within that Pod.
3. Query Prometheus Metrics
Querying Prometheus metrics refers to accessing and retrieving metrics data from a Prometheus server or endpoint. After that, we can use these metrics to obtain real-time information about the PVC capacity and utilization.
The kubelet_volume_stats_available_bytes and kubelet_volume_stats_capacity_bytes are metrics provided by kubelet, the primary node agent running on each node in the Kubernetes cluster. Therefore, we can use them to measure the space left in a persistent volume claim in Kubernetes.
Let’s take a look at the script:
$ cat metrics.sh
#!/bin/bash
# Retrieve available space in bytes
available_bytes=$(curl -s http://kubelet-address:10255/metrics | grep kubelet_volume_stats_available_bytes | grep my-pvc | awk '{print $2}')
# Retrieve total capacity in bytes
capacity_bytes=$(curl -s http://kubelet-address:10255/metrics | grep kubelet_volume_stats_capacity_bytes | grep my-pvc | awk '{print $2}')
# Convert bytes to gigabytes (GB)
available_gb=$(echo "scale=2; $available_bytes / 1024^3" | bc)
capacity_gb=$(echo "scale=2; $capacity_bytes / 1024^3" | bc)
# Display the results
echo "Available space: $available_gb GB"
echo "Total capacity: $capacity_gb GB"
In detail, we use curl to get metrics data for the specified PVC from the kubelet Prometheus endpoint. Subsequently, we employ filtering techniques, including commands like grep and awk, to extract the available space and total capacity values. Then, we proceed to convert these values into gigabytes (GB). Lastly, we present these converted values on the terminal, offering insights into the PVC storage utilization and capacity.
4. Conclusion
In this article, we’ve explored various methods to effectively monitor the remaining space in a Persistent Volume Claim (PVC) within Kubernetes clusters. By leveraging command-line tools such as kubectl, we can inspect the PVC and their associated Pods, enabling us to determine the utilized space and calculate the available storage.
Additionally, we’ve demonstrated how Prometheus metrics can provide real-time insights into PVC capacity and utilization, offering a comprehensive approach to managing storage resources in Kubernetes environments.