1. Overview
In Kubernetes, PersistentVolumeClaims (PVCs) are essential for managing persistent storage. They allow users to request storage resources dynamically, ensuring data persistence beyond the lifecycle of individual pods. However, there are times when users encounter difficulties in deleting PVCs, which can lead to operational challenges and resource management issues.
This tutorial aims to address the common causes and solutions for the issue of being unable to delete PVCs.
2. Pod Dependencies
If there are pods still using the PVC, Kubernetes prevents the deletion of the PVC to avoid data loss. Therefore, to delete a PVC, we should delete all the pods that are using that PVC.
Let’s run a command to see the running pods in the default namespace that are using a PVC named my-pvc:
$ kubectl get pods -o json | jq -r '.items[] | select(.spec.volumes[]?.persistentVolumeClaim.claimName=="my-pvc") | .metadata.name'
"my-pod"
This command retrieves all pods across default namespace in the JSON format and uses jq to filter the JSON output. It selects pods with a volume that has a Persistent Volume Claim named my-pvc and then prints the names of those pods.
From the output, we notice that a pod named my-pod is using the PVC named my-pvc.
Now, if we attempt to delete the PVC without deleting this pod, the PVC becomes stuck in a terminating state:
$ kubectl delete pvc my-pvc
persistentvolumeclaim "my-pvc" deleted
After running this command, we observe that it never terminates.
To avoid this, we delete all the pods that are using my-pvc. Let’s proceed by deleting the pod my-pod, which currently uses my-pvc:
$ kubectl delete pod my-pod
pod "my-pod" deleted
After deleting the pod, we can now delete my-pvc as we attempted previously without it getting stuck.
3. Finalizer Protection
Finalizers are special metadata fields that ensure certain cleanup actions are taken before the object is deleted. If a PVC has a finalizer, it may be preventing the deletion. This often happens when a persistent volume is protected.
Let’s verify whether the PVC named my-pvc in the default namespace is protected or not:
$ kubectl describe pvc my-pvc | grep Finalizers
Finalizers: [kubernetes.io/pvc-protection]
If we see output like this, the PVC is protected. We can resolve this issue in two ways.
3.1. Using kubectl patch
We can fix it directly from the command line by setting the finalizers to null using kubectl patch:
$ kubectl patch pvc my-pvc -p '{"metadata":{"finalizers":null}}'
persistentvolumechain/my-pvc patched
3.2. Editing the PVC Manually
Alternatively, we can manually edit the PVC to remove the finalizers:
$ kubectl edit pvc my-pvc
This command outputs the PVC’s details in JSON format. We look for the metadata section to find the finalizers:
"metadata": {
"finalizers": [
"kubernetes.io/pvc-protection"
]
}
We remove the entire finalizers section from the metadata and save the changes.
After we remove the finalizers using either method, we can delete the PVC that is currently stuck in a terminating state.
4. Namespace Deletion
When we mark a namespace containing PVCs for deletion, without first deleting the PVCs themselves, it can cause the PVCs to remain in a terminating state. This usually occurs because of dependencies and finalizers that require clearing before fully deleting the namespace.
Let’s see an example of a PVC in a namespace other than the default namespace:
$ kubectl get pvc --all-namespaces
NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
my-namespace my-pvc Terminating pvc-20d640c3-9440-43f3-80bc-499648de0c95 100Mi RWO standard <unset> 2m46s
We notice that my-pvc is running in a namespace named my-namespace.
Now let’s mark this namespace for deletion:
$ kubectl delete namespace my-namespace
namespace "my-namespace" deleted
After that, let’s try to delete the my-pvc in my-namespace:
$ kubectl delete pvc my-pvc -n my-namespace
persistentvolumeclaim "my-pvc" deleted
We find that the PVC remains stuck in a terminating state. Even if we try to solve it by removing the finalizers, it gives us an error:
$ kubectl patch pvc my-pvc -n my-namespace -p '{"metadata":{"finalizers":null}}'
Error from server (NotFound): namespaces "my-namespace" not found
To solve this issue, we can recreate the namespace with the same name as it was before, then remove the finalizers, and finally delete the PVC.
5. Conclusion
Deleting PersistentVolumeClaims in Kubernetes can sometimes be challenging due to various factors such as finalizers, pod dependencies, and namespace issues. In this article, we discussed these common issues and their solutions, so we can effectively manage PVC deletions and maintain a smooth Kubernetes environment.