1. Introduction
In the world of Linux, tracking newly created processes is crucial for us as system administrators, developers, and cybersecurity experts. Knowing what processes are running, when they start, and what they’re doing can offer significant insights into our system performance, application behavior, and security posture.
In this tutorial, we’ll shed light on various tools and methods available in Linux for tracking processes, helping us make informed decisions on system monitoring and security. Let’s get started!
2. Why Track Processes?
Understanding which processes are spawned on a system has multiple advantages, particularly for those responsible for system health, security, and performance optimization.
As system administrators, tracking processes helps us in diagnosing resource consumption issues, thereby ensuring that the system runs smoothly. Also, as developers, we may use it to debug applications, ensuring that they initiate and terminate correct processes as expected.
Furthermore, as security professionals, we can benefit from process tracking to monitor unusual or unauthorized activities, providing an additional layer of security against potential threats.
Simply put, tracking processes can help in compliance reporting, audit trails, and troubleshooting. Given these benefits, it’s no wonder that process tracking is an indispensable part of modern Linux system administration and security.
3. Enabling Kernel Features
Before we dive into the different tools available for process tracking, it’s vital to enable certain kernel features.
Specifically, we should make sure that Kernel Dynamic Probes (kprobes) are enabled to facilitate detailed process tracking and ensure BPF Compiler Collection (BCC) and kernel headers are installed.
3.1. Understanding and Enabling Kprobes
Kprobes is a powerful feature in the Linux kernel that allows us to dynamically insert probes (or hooks) into kernel functions or instructions without modifying the kernel’s source code. These probes monitor and trace various kernel and user-space events, including process creation and system calls. Kprobes is invaluable for debugging, performance analysis, and process tracking.
Thus, before we can effectively track newly created processes or perform other kernel-related tracing tasks, we must ensure that kprobes is enabled in the kernel configuration. If kprobes isn’t enabled, we won’t have access to this dynamic probing capability.
Therefore, let’s first check if kprobes is active:
$ grep CONFIG_KPROBE_EVENTS /boot/config-`uname -r`
# CONFIG_KPROBE_EVENTS = y
Here, we use the grep command, a command-line utility that searches for pattern matching within text files and extracts the lines that match a specific pattern or keyword:
- CONFIG_KPROBE_EVENTS – the pattern or keyword we are searching for
- /boot/config-$(uname -r) – the path to a specific directory (/boot) and the beginning of the filename we want to search
- uname -r – retrieves and replaces the backticks with the currently running Linux kernel version
Putting it all together, the command essentially instructs the system to search for lines containing the text “CONFIG_KPROBE_EVENTS” in a file named after the currently running kernel version within the /boot directory.
3.2. Output Scenarios
Now, let’s explore the possible scenarios and explanations for the output.
If the output is “# CONFIG_KPROBE_EVENTS=y,” as we can see in this example, it indicates that kprobes is enabled in the kernel configuration. Therefore, kprobes functionality is available for use in this kernel.
However, if the output is “# CONFIG_KPROBE_EVENTS isn’t set,” it means that kprobes isn’t enabled in the kernel configuration. Consequently, kprobes functionality isn’t available for use in this kernel. If kprobes isn’t enabled in our kernel configuration, but we want to use this powerful feature for process tracking, we’ll need to reconfigure and rebuild the kernel with kprobes support. This process involves the kernel’s configuration and then compiling and installing the new kernel.
Notably, we must understand that kernel configuration and compilation can be complex and require careful attention to our specific Linux distribution and kernel version. We should always consult our distribution documentation or community resources for detailed instructions tailored to different scenarios.
3.3. Installing BCC and Kernel Headers
Additionally, it’s essential to install BCC and kernel headers before using tools like execsnoop and bpftrace for tracking processes.
For Debian-based systems (e.g., Ubuntu), we can install them with the apt package manager:
$ sudo apt install bpfcc-tools linux-headers-$(uname -r)
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
binutils bpfcc-tools-doc cpp cpp-11 cpp-8 dkms fakeroot g++ g++-8 gcc gcc-8 gcc-8-base
libasan5 libatomic1 libc-dev-bin libc6-dev libcc1-0 libcilkrts5 libfakeroot
libgcc-8-dev libitm1 liblsan0 libmpx2 libquadmath0 libstdc++-8-dev libubsan1
linux-headers-5.4.0-91
...
Setting up linux-headers-5.4.0-91 (5.4.0-91.102) ...
Processing triggers for libc-bin (2.31-0ubuntu9.17) ...
Alternatively, for Red Hat-based systems (e.g., CentOS), we can use yum:
$ sudo yum install bpfcc-tools kernel-headers-$(uname -r)
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
* base: centos.mirror.liquidtelecom.com
* epel: centos.mirror.liquidtelecom.com
* extras: centos.mirror.liquidtelecom.com
* updates: centos.mirror.liquidtelecom.com
Resolving Dependencies
--> Running transaction check
---> Package bpfcc-tools.x86_64 0:0.17.0-3.el8 will be installed
---> Package kernel-headers.x86_64 0:4.18.0-348.2.1.el8_5 will be installed
...
Installed:
bpfcc-tools-0.17.0-3.el8.x86_64
kernel-headers-4.18.0-348.2.1.el8_5.x86_64
...
Complete!
In both cases, the apt or yum package manager installs the necessary packages, which include bpfcc-tools for tools like execsnoop and bpftrace, as well as linux-headers or kernel-headers corresponding to the currently running kernel ($(uname -r)). These packages are vital to using Berkeley Packet Filter (BPF) tools and for compiling BPF programs to trace system events effectively.
4. Using execsnoop
execsnoop is a powerful command-line tool that comes as part of BCC. It allows us to track process execution in real-time, offering invaluable data for debugging, monitoring, and security. However, unlike traditional monitoring tools, execsnoop uses BPF to provide a low-overhead, highly efficient tracking mechanism.
To start the tracking process, we can simply run the execsnoop command with administrative privileges (sudo):
$ sudo execsnoop
PCOMM PID PPID RET ARGS
sh 24872 24414 0 /bin/sh -c date
date 24873 24872 0 date
As we can see, it starts monitoring any new processes, printing out details such as the Process ID (PID), Parent Process ID (PPID), Return Value (RET), and Arguments (ARGS):
- PCOMM – represents the parent process’s name or command that initiated the execution of the child process
- PID – shows the unique identifier for each running process
- PPID – displays the Process ID of the parent process that triggered the execution of the child process
- RET – indicates the return value of the executed process
- ARGS – lists the command and its arguments that were executed as a process
For our output, the first entry shows that a shell process with a PID of 24872 (child) was spawned by a parent process with the name “sh” and a PPID of 24414. The return value is 0, indicating success. The executed command was /bin/sh -c date.
Also, the second entry follows a similar process, but the executed command is simply date.
5. System Call Auditing With auditd
Another effective method for keeping tabs on newly created processes is using the Linux Auditing System (auditd). This robust tool plays a pivotal role in generating audit records that detail security-relevant events, allowing us as administrators to maintain a vigilant eye on the system’s activities.
However, auditd isn’t a part of all Linux distributions by default. Thus, we might need to install auditd before we begin.
Afterward, let’s proceed to enable it and start viewing logs.
5.1. Enabling and Viewing Logs
To harness the power of auditd, we must start by enabling it.
First, we enable auditd to start at boot with systemctl:
$ sudo systemctl enable auditd
Created symlink /etc/systemd/system/multi-user.target.wants/auditd.service → /usr/lib/systemd/system/auditd.service.
Then, we start the auditd service:
$ sudo systemctl start auditd
With these commands, auditd monitors system calls and creates a trail of logs that we can later scrutinize for potential security breaches.
5.2. Accessing and Viewing Logs
To access these logs, we navigate to the designated location where the system stores audit records, typically /var/log/audit/audit.log. Then, we employ the tail command.
tail is a command-line utility in Linux for displaying the tail end of files, which is often helpful in monitoring log files in real-time. With the -f flag (for follow or continuous), we enable the command to keep running and display new content as it’s appended to the file, ensuring we stay abreast of system activities as they unfold:
$ tail -f /var/log/audit/audit.log
type=SYSCALL msg=audit(1632567895.125:142): arch=c000003e syscall=59 success=yes exit=0 a0=7ffd869174c0 a1=7ffd86916890 a2=7ffd869169b0 a3=7ffd869179d0 items=2 ppid=22752 pid=22950 auid=1000 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=pts0 ses=13 comm="bash" exe="/usr/bin/bash" key=(null)
type=CWD msg=audit(1632567895.125:142): cwd="/home/user"
type=PATH msg=audit(1632567895.125:142): item=0 name="/bin/bash" inode=12345 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL cap_fp=0000000000000000 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(1632567895.125:142): item=1 name="/lib64/ld-linux-x86-64.so.2" inode=54321 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL cap_fp=0000000000000000 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
As we can see, the command displays entries from the audit.log file in real-time. Each entry contains detailed information about a system call event, including the type of syscall, timestamp, success status, process details, and file-related information.
However, while system-calls auditing with auditd is a powerful ally, we must tread carefully to optimize its effectiveness and mitigate potential drawbacks. First, we should remember that audit logs can accumulate swiftly and consume precious disk space. We can implement a sound log rotation strategy to prevent the logs from overwhelming the storage capacity.
Also, we should safeguard the integrity of our audit logs by ensuring that only authorized personnel can access them. Unauthorized access to these logs could compromise system security, making stringent access controls a top priority.
6. Discovering New Processes with forkstat
forkstat is a lightweight tool that monitors process fork, exec, and other events.
Similar to our previous tools, we can install forkstat with apt or yum:
# For Debian-based systems
$ sudo apt install forkstat
# For Red Hat-based systems
$ sudo yum install forkstat
Then, we can invoke the forkstat command using sudo to launch it and monitor process creation events in real-time:
$ sudo forkstat
2023-09-25 10:15:31 1022 1021 clone flags: 0x3f0
2023-09-25 10:15:31 1024 1025 execve(5234): uname
2023-09-25 10:15:31 1025 1028 exit(5234)
2023-09-25 10:15:31 1022 1024 clone flags: 0x3f0
2023-09-25 10:15:31 1023 1029 execve(5235): ps
2023-09-25 10:15:31 1021 1022 exit(5235)
2023-09-25 10:15:32 1022 1021 clone flags: 0x3f0
2023-09-25 10:15:32 1029 1023 execve(5236): top
2023-09-25 10:15:32 1020 1024 exit(5236)
2023-09-25 10:15:32 1021 1022 clone flags: 0x3f0
2023-09-25 10:15:32 1024 1025 execve(5237): ls
2023-09-25 10:15:32 1027 1022 exit(5237)
2023-09-25 10:15:32 1028 1021 clone flags: 0x3f0
2023-09-25 10:15:32 1029 1022 execve(5238): cat
2023-09-25 10:15:32 1022 1027 exit(5238)
2023-09-25 10:15:32 1024 1028 clone flags: 0x3f0
2023-09-25 10:15:32 1022 1026 execve(5239): gcc
2023-09-25 10:15:32 1028 1021 exit(5239)
^C
--- forkstat summary ---
13499 forks
0 forks where parent was waiting
0 forks w/ parent in uninterruptible sleep
1 clone calls
0 clone parent/child race
0 fork or clone races
0 signals delivered
0 voluntary context switches
1 involuntary context switches
Let’s understand our output columns:
- Timestamp – shows the timestamp of each event, indicating when it occurred
- PID – the first numeric column displays the PID of the process generating the event
- PPID – the second numeric column is the parent PID
- Process Name – represents the process name associated with the event, if available
- Event Description – describes the event that occurred
As for the event descriptions, here are some common event types:
- clone – indicates the creation of a new process or thread
- execve – signifies the execution of a new program
- exit – shows when a process or thread exits
- event details – shows the details of the event in parentheses after the event description
Lastly, after the event listings, there’s a summary section (when we terminate forkstat by pressing Ctrl+C) that provides statistics about various activities.
7. Conclusion
In this article, we traversed various methods for tracking newly created processes in Linux environments. By leveraging specialized tools like execsnoop, auditd, and forkstat, we have demonstrated the importance of having a nuanced approach to process monitoring. Whether we require high-level summaries or low-level details, we have discussed many tools that Linux offers to meet our needs.
Finally, effective process tracking isn’t just a technical requirement but also a critical aspect of maintaining a secure and performant system. As system architectures evolve, so do the methods for tracking processes, making it vital to stay updated and adaptable.