1. Introduction
In this tutorial, we’ll delve into the methods for listing recently deleted pods in Kubernetes. First, we’ll review how the lifecycle of a pod works. Then, we’ll understand some commands for listing recently deleted pods. Finally, we’ll see an example of how to run these commands in a Kubernetes environment.
2. Exploring Kubernetes Pod Lifecycle
Before learning how to list deleted pods, it’s essential to understand the lifecycle of a Kubernetes pod. A pod is the smallest and simplest unit that can be created and managed in Kubernetes. It encapsulates one or more closely associated containers, storage resources, a unique network IP, and settings defining how the containers should operate. Thus, a pod represents a singular instance of an application.
For better management, pods go through various phases in their lifecycle. Each one gives us an overview of their condition at that time. The phases are:
Phase
Description
Pending
The initial phase of a pod. During this phase, Kubernetes assigns a node to run the pod based on specifications. This is also where the system downloads and initializes the container images.
Running
The system has bound the node to the pod and created all containers with at least one run. Here, the system monitors the health of the running containers via readiness and liveness probes.
Succeeded
This appears after successful completion, indicating that all containers have successfully closed and won’t restart. As such, consider pods in this state terminated, as they’re no longer actively running.
Failed
All containers were terminated, and at least one container was terminated in failure.
Unknown
In some cases, Kubernetes may not be able to obtain the state of a pod, usually due to communication problems with the node. When this happens, the system assigns the pod the status unknown.
It’s important to note that pods are only programmed once during their lifetime. In other words, once created on the intended node, it remains until finished or deleted. If any changes or updates are necessary for the pod, the controller will create new pods based on the updated model to replace the existing ones.
Similarly, the system never relocates a pod to a different node. If necessary, the system replaces the original pod with a new one that’s almost identical, even retaining the same name if it’s important, but its UID will be different.
3. Listing Recently Deleted Pods
Given the characteristics of a pod’s lifecycle, it’s not possible to directly access information about deleted pods using standard commands. However, we can take an approach that uses the Kubernetes event system.
Kubernetes generates events to signal various changes in the pod lifecycle. With this, we can check some important information about recently deleted pods, such as their namespaces and deletion timestamps.
With this approach, we’ll get a list of recently deleted pods. However, this list will only include pods that the system deleted within the last hour. This is the default TTL for events in Kubernetes, so this is also the time they’re stored.
To get a list with only the names of the deleted pods, we can run the following command:
$ kubectl get event -o custom-columns=NAME:.metadata.name | cut -d "." -f1
In it, we’re formatting the output of the events using -o custom-columns=NAME:.metadata.name. In other words, we’re creating a column called NAME and extracting the value of the deleted pod’s metadata.name field to fill that column.
To obtain a list of common information about the pod’s event, we can use the command:
$ kubectl get events --field-selector reason=Killing --sort-by='.metadata.creationTimestamp'
Using the –field-selector reason=Killing argument, we filter the events to return. So, we chose the reason Killing, which indicates that we’re interested in events related to the finalization of pods.
For a better view, we sort the returned list based on the metadata creation timestamp. To do this, we use the argument –sort-by=’.metadata.creationTimestamp. This means listing the pods chronologically, from oldest to newest.
These two commands provide basic information about the deleted pod. Note that if the pod has been deleted for more than 60 minutes, these commands won’t return the requested information.
4. Example
Let’s now run the commands presented in a Kubernetes environment and analyze the output. Then, we need to have a pod created and recently deleted.
4.1. Creating a Pod
First, we need to create a pod:
$ kubectl run my-pod --image=nginx
In this command, we create a pod named my-pod to run an Nginx container. This follows the imperative creation format, where a command directly triggers an action in Kubernetes.
The other creation approach is declarative, where a configuration file (YAML or JSON), also called a manifest, defines the properties of the pod. After interpreting the file, Kubernetes creates the pod based on the specified settings.
Since our focus is only on listing a recently deleted pod, we won’t go into other specific details.
4.2. List Pods
To verify that the system has created our pod, we can list all pods in the cluster using the following command:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
my-pod 1/1 Running 0 13s
nginx 1/1 Running 0 10h
As we can see, our pod is running successfully.
4.3. Deleting a Pod
Now, let’s delete our pod:
$ kubectl delete pod my-pod
To check if the system has deleted our pod, we can list all the pods again. This will show that it’s not on the list of active pods:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx 1/1 Running 0 10h
4.4. Listing Deleted Pods
As explained earlier, one of the ways to list deleted pods is by filtering only their name (or the selected metadata):
$ kubectl get event -o custom-columns=NAME:.metadata.name | cut -d "." -f1
NAME
my-pod
my-pod
my-pod
my-pod
my-pod
my-pod
In this example, the deleted pod generated 6 events. While we’re filtering, we only have access to the pod’s name and no other information.
To view more information, we can run:
$ kubectl get events --field-selector reason=Killing --sort-by='.metadata.creationTimestamp'
LAST SEEN TYPE REASON OBJECT MESSAGE
20m Normal Scheduled pod/my-pod Successfully assigned default/my-pod to kubernetes-n2
20m Normal Pulling pod/my-pod Pulling image "nginx"
19m Normal Pulled pod/my-pod Successfully pulled image "nginx" in 8.108s (8.108s including waiting)
19m Normal Created pod/my-pod Created container my-pod
19m Normal Started pod/my-pod Started container my-pod
17m Normal Killing pod/my-pod Stopping container my-pod
With this command, we can trace events from pod creation to termination. In this simple scenario, the system displays only normal events. A broader range of events and types may appear in complex environments (such as production settings).
5. Conclusion
In this article, we’ve studied the process of listing deleted pods in Kubernetes. As we’ve learned, we need to use the Kubernetes event system to find out about these pods. With this, we can list and analyze deletion events.