1. Overview
When we list the processes in a system, the existence of processes like ksoftirqd/n, where n is a non-negative integer, may attract our attention. Additionally, we may sometimes observe that these processes consume a high amount of CPU, thus impacting the system performance.
In this tutorial, we’ll discuss what these ksoftirqd processes are and why they sometimes consume high CPU. We’ll also generate a case where the CPU consumption of these processes increases.
2. What Are ksoftirqd Processes?
Software interrupts are generated by the kernel, whereas hardware interrupts are generated by hardware. Disk I/O, network processing, and task scheduling are just a few examples of when software interrupts are triggered.
Sometimes, software interrupts might be triggered much more quickly than they can be handled. For example, a heavy load of software interrupts can be triggered when a network interface card receives a huge number of packets in a short time.
Software interrupts are generally less time-critical than hardware interrupts. Therefore, in the case of a heavy software interrupt load, the kernel queues those interrupts so that they can be handled later.
ksoftirqd daemons are responsible for handling these deferred software interrupts but in the process context. So, their CPU consumption might increase depending on the load.
There are generally multiple ksoftirqd processes in a system since ksoftirqd processes are per-CPU kernel threads. So, if we list the processes in a system, we can see ksoftirqd/n entries in the output, where n is the number of the CPU the ksoftirqd process runs on.
3. An Example of Generating a High CPU Consumption
We’ll see an example of increasing CPU usage of ksoftirqd processes in this section.
We’ll use two hosts, host1 and host2, in our example*.* Their IP addresses are 10.0.2.5 and 10.0.2.6, respectively.
We’ll try to increase the CPU usage of the ksoftirqd processes in host1 by starting a TCP SYN flood attack from host2.
3.1. The CPU Usage of ksoftirqd Processes Before the Attack
We’ll check the CPU usage of the ksoftirqd processes running on host1 before starting the attack. Firstly, let’s find the PIDs of these processes:
$ ps -e | grep ksoftirqd | grep -v grep
13 ? 00:00:00 ksoftirqd/0
19 ? 00:00:00 ksoftirqd/1
We use the ps command for listing the processes on host1. The -e option of ps selects all processes. However, we filter the output using two grep commands, grep ksoftirqd and grep -v grep, to show only the ksoftirqd processes.
There are two ksoftirqd processes on host1. The PIDs of ksoftirqd/0 and ksoftirqd/1 are 13 and 19, respectively. ksoftirqd/0 is the process that runs on CPU 0. Similarly, ksoftirqd/1 is the process that runs on CPU 1.
There are two ksoftirqd processes since host1 has two CPUs:
$ nproc
2
The nproc command prints the number of processing units available.
Having determined the PIDs of the ksoftirqd processes, let’s check the CPU usage of these processes using the top command:
$ top -p 13,19
top - 00:20:11 up 1 min, 1 user, load average: 0.65, 0.22, 0.08
Tasks: 2 total, 0 running, 2 sleeping, 0 stopped, 0 zombie
%Cpu(s): 3.6 us, 3.6 sy, 0.0 ni, 92.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 1775.1 total, 78.6 free, 1004.8 used, 691.7 buff/cache
MiB Swap: 2096.0 total, 2094.2 free, 1.8 used. 573.8 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
13 root 20 0 0 0 0 S 0.0 0.0 0:00.01 ksoftirqd/0
19 root 20 0 0 0 0 S 0.0 0.0 0:00.02 ksoftirqd/1
The -p option of top monitors only the specified processes. The CPU usage of both processes shown in the %CPU column is 0.0%.
3.2. Starting a TCP Server on host1
Let’s start a TCP server on host1 using the nc command:
$ nc -lv 1234
Ncat: Version 7.91 ( https://nmap.org/ncat )
Ncat: Listening on :::1234
Ncat: Listening on 0.0.0.0:1234
By default, nc uses TCP as the communication protocol. The -l option tells nc to listen to a specific port for incoming connections. 1234 is the port number. Additionally, the -v option provides a more verbose output.
3.3. Starting a SYN Flood Attack on host2
hping3 is a network tool that can send custom TCP/IP packets and display the replies. We’ll use hping3 for starting a SYN flood attack from host2 to host1:
$ sudo hping3 10.0.2.5 --flood -S --rand-source -p 1234
HPING 10.0.2.5 (enp0s3 10.0.2.5): S set, 40 headers + 0 data bytes
hping in flood mode, no replies will be shown
We need root privileges to use hping3, so we use it together with the sudo command.
10.0.2.5 is the IP address of the target. It’s the IP address of host1 in our example.
The –flood option of hping3 sends the packets as fast as possible without waiting for the replies.
The -S option specifies to send SYN packets. SYN is the first packet sent from a client to a server to establish a TCP connection.
The –rand-source option enables sending packets with random source addresses.
The -p option specifies the destination port. This is 1234 in our case, as the nc server on host1 listens to port 1234 for incoming connections.
3.4. The CPU Usage of ksoftirqd Processes During the Attack
Having started the attack from host2 to host1, let’s check the CPU usage of the ksoftirqd processes in the output of the top command in host1 once more:
top - 00:25:36 up 6 min, 1 user, load average: 0.78, 0.31, 0.14
Tasks: 2 total, 1 running, 1 sleeping, 0 stopped, 0 zombie
%Cpu(s): 5.0 us, 0.0 sy, 0.0 ni, 60.0 id, 0.0 wa, 5.0 hi, 30.0 si, 0.0 st
MiB Mem : 1775.1 total, 93.0 free, 1023.9 used, 658.2 buff/cache
MiB Swap: 2096.0 total, 2093.0 free, 3.0 used. 559.6 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
19 root 20 0 0 0 0 R 100.0 0.0 1:00.17 ksoftirqd/1
13 root 20 0 0 0 0 S 0.0 0.0 0:00.01 ksoftirqd/0
The CPU consumption of ksoftirqd/1 in the above snapshot is 100.0%, while the CPU consumption of ksoftirqd/0 is 0.0%.
Depending on the system and type of load, only one CPU might handle the software interrupts, or the software interrupts can be spread between the CPUs. In our case, the ksoftirqd/1 process seems to handle the software interrupts. However, the opposite is also possible, i.e., ksoftirqd/0 consuming high CPU.
4. Conclusion
In this article, we discussed what ksoftirqd processes are and why they sometimes consume high CPU.
Firstly, we learned that ksoftirqd processes are per-CPU kernel threads. When there is a high number of software interrupts, the kernel queues these software interrupts for deferred processing since software interrupts aren’t generally as time-critical as hardware interrupts. We learned that ksoftirqd daemons handle these deferred software interrupts.
Then, we experimented with the high CPU consumption of ksoftirqd processes by starting a TCP SYN attack from one host to another host. We saw that the CPU consumption of one of the ksoftirqd processes on the attacked host increased.