1. Overview

In Kubernetes, PersistentVolumeClaims (PVCs) are essential for managing persistent storage. They allow users to request storage resources dynamically, ensuring data persistence beyond the lifecycle of individual pods. However, there are times when users encounter difficulties in deleting PVCs, which can lead to operational challenges and resource management issues.

This tutorial aims to address the common causes and solutions for the issue of being unable to delete PVCs.

2. Pod Dependencies

If there are pods still using the PVC, Kubernetes prevents the deletion of the PVC to avoid data loss. Therefore, to delete a PVC, we should delete all the pods that are using that PVC.

Let’s run a command to see the running pods in the default namespace that are using a PVC named my-pvc:

$ kubectl get pods -o json | jq -r '.items[] | select(.spec.volumes[]?.persistentVolumeClaim.claimName=="my-pvc") | .metadata.name'

"my-pod"

This command retrieves all pods across default namespace in the JSON format and uses jq to filter the JSON output. It selects pods with a volume that has a Persistent Volume Claim named my-pvc and then prints the names of those pods.

From the output, we notice that a pod named my-pod is using the PVC named my-pvc.

Now, if we attempt to delete the PVC without deleting this pod, the PVC becomes stuck in a terminating state:

$ kubectl delete pvc my-pvc

persistentvolumeclaim "my-pvc" deleted

After running this command, we observe that it never terminates.

To avoid this, we delete all the pods that are using my-pvc. Let’s proceed by deleting the pod my-pod, which currently uses my-pvc:

$ kubectl delete pod my-pod

pod "my-pod" deleted

After deleting the pod, we can now delete my-pvc as we attempted previously without it getting stuck.

3. Finalizer Protection

Finalizers are special metadata fields that ensure certain cleanup actions are taken before the object is deleted. If a PVC has a finalizer, it may be preventing the deletion. This often happens when a persistent volume is protected.

Let’s verify whether the PVC named my-pvc in the default namespace is protected or not:

$ kubectl describe pvc my-pvc | grep Finalizers

Finalizers:    [kubernetes.io/pvc-protection]

If we see output like this, the PVC is protected. We can resolve this issue in two ways.

3.1. Using kubectl patch

We can fix it directly from the command line by setting the finalizers to null using kubectl patch:

$ kubectl patch pvc my-pvc -p '{"metadata":{"finalizers":null}}'

persistentvolumechain/my-pvc patched

3.2. Editing the PVC Manually

Alternatively, we can manually edit the PVC to remove the finalizers:

$ kubectl edit pvc my-pvc

This command outputs the PVC’s details in JSON format. We look for the metadata section to find the finalizers:

"metadata": {
  "finalizers": [
    "kubernetes.io/pvc-protection"
  ]
}

We remove the entire finalizers section from the metadata and save the changes.

After we remove the finalizers using either method, we can delete the PVC that is currently stuck in a terminating state.

4. Namespace Deletion

When we mark a namespace containing PVCs for deletion, without first deleting the PVCs themselves, it can cause the PVCs to remain in a terminating state. This usually occurs because of dependencies and finalizers that require clearing before fully deleting the namespace.

Let’s see an example of a PVC in a namespace other than the default namespace:

$ kubectl get pvc --all-namespaces  

NAMESPACE     NAME    STATUS       VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
my-namespace  my-pvc  Terminating  pvc-20d640c3-9440-43f3-80bc-499648de0c95   100Mi      RWO            standard       <unset>                 2m46s

We notice that my-pvc is running in a namespace named my-namespace.

Now let’s mark this namespace for deletion:

$  kubectl delete namespace my-namespace

namespace "my-namespace" deleted

After that, let’s try to delete the my-pvc in my-namespace:

$ kubectl delete pvc my-pvc -n my-namespace

persistentvolumeclaim "my-pvc" deleted

We find that the PVC remains stuck in a terminating state. Even if we try to solve it by removing the finalizers, it gives us an error:

$ kubectl patch pvc my-pvc -n my-namespace -p '{"metadata":{"finalizers":null}}' 

Error from server (NotFound): namespaces "my-namespace" not found

To solve this issue, we can recreate the namespace with the same name as it was before, then remove the finalizers, and finally delete the PVC.

5. Conclusion

Deleting PersistentVolumeClaims in Kubernetes can sometimes be challenging due to various factors such as finalizers, pod dependencies, and namespace issues. In this article, we discussed these common issues and their solutions, so we can effectively manage PVC deletions and maintain a smooth Kubernetes environment.