1. Overview
Kubernetes, the popular container orchestration platform, offers various commands to inspect and manage pods, the smallest deployable units in a Kubernetes cluster. In the dynamic world of DevOps, understanding the contents of files within these pods is crucial for effective troubleshooting and maintenance. However, quickly and comprehensively inspecting these file contents can be challenging.
Although Kubernetes does not have a direct command for listing files in a pod, there are several approaches to achieve this objective. Therefore, in this tutorial, we will explore these methods, providing practical insights for inspecting Kubernetes pods.
2. Inspection of Pods
When inspecting a pod to check its files, the primary focus is on accessing and examining the file systems within the containers that make up the pod. Additionally, a pod in Kubernetes can host one or more containers, each with its isolated file system. Consequently, inspecting a pod involves navigating through these containers to verify their files’ presence, structure, and content.
3. kubectl exec Command
The kubectl exec command allows users to execute commands directly inside a container within a pod. First of all, let’s get all the pods that are running in our system:
$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default sys-pod 1/1 Running 0 3m32s
kube-system coredns-7db6d8ff4d-g9gjw 1/1 Running 5 (27m ago) 15d
kube-system etcd-minikube 1/1 Running 5 (27m ago) 15d
kube-system kube-apiserver-minikube 1/1 Running 5 (27m ago) 15d
kube-system kube-controller-manager-minikube 1/1 Running 5 (27m ago) 15d
kube-system kube-proxy-w5v7w 1/1 Running 5 (27m ago) 15d
kube-system kube-scheduler-minikube 1/1 Running 5 (27m ago) 15d
kube-system storage-provisioner 1/1 Running 13 (27m ago) 15d
From the output, we can see several pods are running. One of the running pods is sys-pod. When inspecting a pod, by default, it inspects the first container it has. Let’s inspect the sys-pod now to check its contents:
$ kubectl exec sys-pod -- ls -la /
total 48
drwxr-xr-x 1 root root 4096 Jul 23 06:56 .
drwxr-xr-x 1 root root 4096 Jul 23 06:56 ..
-rwxr-xr-x 1 root root 0 Jul 23 06:56 .dockerenv
drwxr-xr-x 2 root root 12288 May 18 2023 bin
drwxr-xr-x 5 root root 360 Jul 23 06:56 dev
drwxr-xr-x 1 root root 4096 Jul 23 06:56 etc
drwxr-xr-x 2 nobody nobody 4096 May 18 2023 home
drwxr-xr-x 2 root root 4096 May 18 2023 lib
lrwxrwxrwx 1 root root 3 May 18 2023 lib64 -> lib
dr-xr-xr-x 262 root root 0 Jul 23 06:56 proc
drwx------ 2 root root 4096 May 18 2023 root
dr-xr-xr-x 13 root root 0 Jul 23 06:56 sys
drwxrwxrwt 2 root root 4096 May 18 2023 tmp
drwxr-xr-x 4 root root 4096 May 18 2023 usr
drwxr-xr-x 1 root root 4096 Jul 23 06:56 var
The command outputs the contents of the root (/) folder in sys-pod. Moreover, if we want to inspect a different folder instead of the root folder, we can specify it in the command after ls -la. From the previous command, we see some folder names in the root folder. Let’s check the contents of the sys folder:
$ kubectl exec sys-pod -- ls -la /sys
total 4
dr-xr-xr-x 13 root root 0 Jul 23 06:56 .
drwxr-xr-x 1 root root 4096 Jul 23 06:56 ..
drwxr-xr-x 2 root root 0 Jul 23 06:57 block
drwxr-xr-x 40 root root 0 Jul 23 06:57 bus
drwxr-xr-x 54 root root 0 Jul 23 06:57 class
drwxr-xr-x 4 root root 0 Jul 23 06:57 dev
drwxr-xr-x 15 root root 0 Jul 23 06:56 devices
drwxrwxrwt 2 root root 40 Jul 23 06:56 firmware
drwxr-xr-x 7 root root 0 Jul 23 06:56 fs
drwxr-xr-x 2 root root 0 Jul 23 06:57 hypervisor
drwxr-xr-x 17 root root 0 Jul 23 06:56 kernel
drwxr-xr-x 184 root root 0 Jul 23 06:57 module
drwxr-xr-x 3 root root 0 Jul 23 06:57 power
In the input command, we specified /sys to see its contents. This command then outputs the contents of the sys folder present in the sys-pod.
4. Accessing Container Filesystem from the Host
The earlier method is ineffective if the pod’s container lacks the ls command, as is typical for many pods in the kube-system namespace. Consequently, this approach won’t be suitable for the majority of container images in real-world Kubernetes clusters. Therefore, let’s explore a more universally applicable solution.
4.1. Retrieve Container ID from Pod
From section 3, we saw different pod names, including kube-apiserver-minikube in the kube-system namespace. First, let’s get the container ID from the kube-apiserver-minikube pod:
$ kubectl get pod kube-apiserver-minikube -n kube-system -o jsonpath='{.status.containerStatuses[0].containerID}'
docker://70d228676fdc03f0767a1161f56dfd1ad9194f9b7512663bd2e451e8ea502276
This command retrieves the container ID of the first container in the kube-apiserver-minikube pod from the kube-system namespace in JSON format. The container ID is 70d228676fdc03f0767a1161f56dfd1ad9194f9b7512663bd2e451e8ea502276.
4.2. Identify Node Hosting the Pod
Secondly, we need to identify the node hosting the pod. In Kubernetes, a node is a physical or virtual machine that runs one or more pods. Let’s find the node where our pod is running:
$ kubectl get pod kube-apiserver-minikube -n kube-system -o jsonpath='{.spec.nodeName}'
minikube
This command outputs the node, which is minikube in our example.
4.3. Establish SSH Connection to the Node
Thirdly, we establish an SSH connection to the minikube node for direct access. Let’s create the connection:
$ minikube ssh
docker@minikube:
4.4. Obtain the Host PID of the Container
Afterwards, we retrieve the PID of the container to interact with the container’s process directly. Let’s find the PID of the container:
docker@minikube:~$ sudo docker inspect --format '{{ .State.Pid }}' 70d228676fdc03f0767a1161f56dfd1ad9194f9b7512663bd2e451e8ea502276
1893
This command outputs the PID of the container, which is 1893.
4.5. Access Container’s Root Filesystem Directory
Finally, we can access the container’s root filesystem directory:
docker@minikube:~$ cd /proc/1893/root
Here, in the input command, 1893 represents the Process ID (PID) of the container’s main process, which we obtained previously.
Once we are in the /proc/1893/root directory, we can list its contents to view the structure of the container’s root filesystem:
docker@minikube:/proc/1893/root:~$ ls
bin boot dev etc go-runner home lib proc root run sbin sys tmp usr var
The output displays the directories and files present in the container’s root filesystem.
Overall, this method provides a more robust way to access the container filesystem from the host.
5. Shell or Bash Usage
An interactive shell can be a convenient way to navigate the files in the file system of a pod. Accessing a shell within one of the containers in the pod allows for the use of various commands to inspect and manage the pod’s file system.
Let’s start an interactive shell session with the sys-pod:
$ kubectl exec -it sys-pod -- sh
/ #
Once inside the shell, we can use various commands. Let’s use lsthe command to see the contents of sys-pod:
$ kubectl exec -it sys-pod -- sh
/ # ls
bin dev etc home lib lib64 proc root sys tmp usr var
The output displays the directories present in the root directory of sys-pod.
Moreover, to start an interactive shell in a specific namespace, we use the namespace flag. For example, let’s use a command to access a shell in one of the pods, etcd-minikube in the kube-system namespace:
$ kubectl exec -it --namespace kube-system etcd-minikube -- sh
/ #
After starting the interactive shell, we can use commands like before to inspect the contents of the etcd-minikube pod.
6. Conclusion
In this article, we have discussed different approaches for inspecting pods running in Kubernetes. For instance, we discussed about using kubectl exec for direct command execution, accessing the container filesystem from the host for more comprehensive exploration, and leveraging interactive shells. These approaches enable efficient and effective management of Kubernetes environments.