1. Introduction
There isn’t always a way to clean up zombie processes. Yet, almost any process can become a zombie.
In this tutorial, we explore what happens when it’s a child process of the one that can’t – the initialization process (init). First, we look at how init can end up as the parent of a zombie. Next, we delve into the mechanics of initialized zombies. Finally, we discuss ways to manually resolve zombies that are direct children of the initialization process.
We tested the code in this tutorial on Debian 11 (Bullseye) with GNU Bash 5.1.4. It should work in most POSIX-compliant environments.
2. Initialized Zombies
Whether it’s systemd, System V (SysV), or another option, Linux needs a mechanism to start itself. As the more modern alternative, let’s look at how systemd can spawn a process that goes on to transition to a zombie (Z) state.
For brevity, despite their differences, we use the classic SysV name init alongside systemd interchangeably to describe the initialization process.
2.1. init as Parent Process
Regardless of the run level, we can have a basic set of kernel modules and packages which run at the start of the operating system (OS). Such basic programs are usually leveraged by processes, which spawn as a fork of PID 1:
$ pstree
systemd─┬─blkmapd
├─cron
├─dbus-daemon
├─login───bash
├─nmbd
├─rpcbind
├─rsyslogd───3*[{rsyslogd}]
├─smbd─┬─cleanupd
│ ├─lpqd
│ └─smbd-notifyd
├─sshd───sshd───sshd───bash───su───bash───su───bash───+
├─2*[systemd───(sd-pam)]
├─systemd-journal
├─systemd-logind
├─systemd-timesyn───{systemd-timesyn}
└─systemd-udevd
Here, we use pstree to see many processes directly stemming from the systemd initialization process:
- filesystem services like blkmapd and smbd
- scheduling services like cron
- message busses like dbus-daemon
- login and terminal generation processes like systemd-logind and login
- low-level name-resolution services like nmbd
- Remote Procedure Call (RPC) handlers like rpcbind
- logging services like rsyslogd and systemd-journal
- remote access services like sshd
- time synchronization services like systemd-timesyn
- device managers like systemd-udevd
Of course, any of the above is a candidate for becoming an initialized zombie, i.e., a zombie that has the initialization process as its parent.
2.2. Process to Zombie
Processes turn into zombies by being left by their parents even after their work as a child is done.
In particular, to release completed children and avoid zombies, the parent executes the wait() system call, usually when handling SIGCHLD.
*When the parent doesn’t acknowledge their completion, children remain in the zombie (Z,
3. Handling Initialization Zombies
Usually, we have two main options to handle zombie processes:
- send SIGCHLD to the parent
- kill the parent
In fact, we might be able to work around cases when even kill -9 doesn’t appear to work for the second case.
However, the situation is complicated when the parent is the initialization process, which:
- ignores unexpected SIGCHLD signals
- can’t be killed without rebooting
Still, one of the functions of systemd and init is to poll the execution status of its children and wait() for them. Since the initialization process usually becomes the new parent of orphan processes, they often get resolved instead of becoming zombies.
Yet, sometimes the mechanism fails and results in a
4. Manually Resolving Initialization Zombies
When init is the parent of a zombie, we don’t have much choice. There are several ways to attempt and resolve the situation:
- ignore the process
- restart init
- reboot
Ignoring zombies might be acceptable when there is a limited number of such processes, and they don’t take up resources or spawn constantly. However, in some cases, despite their definition, zombies can continue holding a resource, preventing a new process from using it.
Also, depending on the initialization process, we might be able to restart it with different tools:
- telinit for SysV provides telinit u for init restarts
- systemctl for systemd provides systemctl daemon-reexec for systemd restarts
Despite not requiring a reboot, such commands may still have unwanted consequences, and whether their benefit outweighs any negative side effects depends on the situation.
Finally, as a last resort, a clean reboot or shutdown plus a new start should solve the problem.
5. Summary
In this article, we talked about our options when an initialization process is the parent of a zombie.
In conclusion, we have ways of handling initialized zombies, but they are limited, and the root cause of the issue should be of primary concern.