1. Overview
Sometimes administrators or security specialists that monitor a network may find that servers are sending unexpected UDP packets. Further, to identify these packets, we need to find the processes that are generating them. However, this isn’t as easy as it might seem, because UDP is stateless. So, to use typical commands that we could apply to TCP connections such as lsof or netstat, we need to enter the command at the exact time the UDP packet is being sent, which is almost impossible to do.
In this tutorial, we’re going to explore two ways in which we can identify the processes that are generating the UDP traffic.
2. Linux Audit Service
The first way to find the PID of the processes sending the UDP traffic is by using the Linux Audit Service.
The Linux Audit Service is a component that allows us to monitor the system. To do this, we add capture rules with criteria, so the service can monitor and record the data. After that, we can search through the recorded data to find out what we want.
Indeed, there are several components to the Linux Audit Service:
- auditd is the daemon
- auditctl controls the daemon and rules
- ausearch enables searching through the recorded data
So, let’s install and use auditd.
2.1. Installation
First, we check for the service by running auditctl:
$ auditctl -l
No rules
In this case, the -l flag shows the current ruleset. So far, there are no rules.
If auditd isn’t installed, we’d get a command not found error when running auditctl.
So, to install auditd on RHEL or CentOS, we’ll use the yum package manager:
$ sudo yum install audit
On the other hand, to install auditd on Ubuntu, we’ll use the apt package manager:
$ sudo apt install auditd
Next, let’s see how we can use the service to monitor UDP traffic.
2.2. Add a Rule
Before using auditd to monitor UDP traffic, we need to add a rule:
$ auditctl -a exit,always -F arch=b64 -F a0=2 -F a1\&=2 -S socket -k SOCKET
Let’s explore this command to understand the meaning of each part:
- -a adds a new rule
- -F adds the conditions
- -S identifies the function or system call we want to audit
- arch=b64 means we are using a 64-bit architecture
- a0 and a1 are the first and second parameters of the socket system call, both include 2 in the case of UDP
- -k makes it easier to search through the logs by adding a key
The bitwise operator near a1 prevents the loss of other values that change the behavior of the system call.
Next, we’ll try sending a UDP request and looking for it in the logs.
2.3. Search Through the Records
Now, let’s call the ping command that sends a DNS lookup request over UDP:
$ ping www.google.com
Next, let’s use the ausearch tool to view the data recorded for our rule:
$ ausearch -i -ts recent -k SOCKET
Further, we can use the grep command to make it easier to spot our ping command:
$ ausearch -i -ts recent -k SOCKET | grep ping
The -i option interprets or transforms some of the fields. For example, -i converts the UID into the corresponding user. The -ts option is the start time for the data we want to see. Here, the recent keyword means 10 minutes.
According to the system, the output might vary a bit, but it should contain some common data:
type=SYSCALL msg=audit(04/23/2023 17:59:00.104:130174) :
arch=x86_64 syscall=socket success=yes exit=3 a0=inet a1=SOCK_RAW a2=icmp a3=0x7ffdf249264c items=0
ppid=29280 pid=32725 auid=root uid=root gid=root euid=root
suid=root fsuid=root egid=root sgid=root fsgid=root tty=pts0 ses=6115
comm=ping exe=/usr/bin/ping
subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=SOCKET2
Here, we inserted newlines for legibility. We can see that auditd captured the UDP requests. Let’s have a look at some of the data that can help identify their source:
- pid is the process ID that was responsible for sending the request
- PPID is the parent process ID, which can be useful in case the command was called from inside a script
- uid is the ID of the user that sent the request
- comm is the command that sent the request
- exe is the command executable location
To delete the rule we previously added, we use the same command but replace -a with -d:
$ auditctl -d exit,always -F arch=b64 -F a0=2 -F a1\&=2 -S socket -k SOCKET
In the next part, we’ll explore another tool to monitor UDP traffic.
3. Systemtap
Another tool that we can use to monitor UDP traffic is Systemtap or stap.
Systemtap is a tool available in Linux that makes it possible to monitor and modify kernel activities without having to change the kernel code itself. It consists of a scripting language as well as a CLI. The scripts written in the Systemtap scripting language are inserted into the kernel after being compiled into C-code kernel modules.
3.1. Installation
First, we’ll make sure we’ve installed Systemtap on the machine:
$ stap -V
If the stap -V command produces an error, then we need to first install Systemtap.
To install Systemtap for Ubuntu, we’ll again use apt:
$ sudo apt-get install -y systemtap gcc
Similarly, to install Systemtap for CentOS/RHEL, we’ll use yum:
$ sudo yum install systemtap systemtap-runtime
$ sudo yum install kernel-debuginfo kernel-debuginfo-common
Importantly, to use certain tapsets, we need to install the kernel debug symbols. How we do this depends on the exact distribution.
To install the kernel debug symbols for Ubuntu, we need to get the correct sources and install the relevant packages:
$ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C8CAB6595FDFF622
$ codename=$(lsb_release -c | awk '{print $2}')
$ sudo tee /etc/apt/sources.list.d/ddebs.list << EOF
deb http://ddebs.ubuntu.com/ ${codename} main restricted universe multiverse
deb http://ddebs.ubuntu.com/ ${codename}-security main restricted universe multiverse
deb http://ddebs.ubuntu.com/ ${codename}-updates main restricted universe multiverse
deb http://ddebs.ubuntu.com/ ${codename}-proposed main restricted universe multiverse
EOF
$ sudo apt-get update
$ sudo apt-get install linux-image-$(uname -r)-dbgsym
To install the kernel debug symbols for CentOS/RHEL, we should be able to use a single command:
$ sudo yum --enablerepo=base-debuginfo install -y kernel-debuginfo-$(uname -r)
Next, let’s see how to use Systemtap to monitor the UDP traffic.
3.2. Running a Systemtap Script
Let’s create a udp_monitor.stp file as the script to monitor UDP traffic using Systemtap:
$ cat udp_monitor.stp
probe udp.sendmsg {
printf ("(%s) with PID: %5d sent UDP to %15s port %d\n", execname(), pid(), daddr, dport)
}
We use the probe keyword to tell Systemtap which tapset to use. So, we’ll be using the udp.sendmsg tapset to monitor UDP traffic.
Then, when a process sends a UDP request, we print several fields:
- execname() – name of the source executable
- pid() – process ID of the source executable
- daddr – destination IP address
- dport – destination port
Next, we run the script to start monitoring UDP traffic:
$ stap -v udp_monitor.stp
Pass 1: parsed user script and 475 library scripts using 272156virt/69388res/3492shr/65956data kb, in 550usr/80sys/635real ms.
Pass 2: analyzed script: 4 probes, 13 functions, 7 embeds, 3 globals using 431592virt/229936res/4740shr/225392data kb, in 1590usr/1710sys/3301real ms.
Pass 3: using cached /root/.systemtap/cache/1e/stap_1eac85c480fa3920fda429fabe71b526_22654.c
Pass 4: using cached /root/.systemtap/cache/1e/stap_1eac85c480fa3920fda429fabe71b526_22654.ko
Pass 5: starting run.
(coredns) with PID: 3494 sent UDP to 8.8.8.8 port 53
(coredns) with PID: 3494 sent UDP to 8.8.8.8 port 53
(coredns) with PID: 3494 sent UDP to 8.8.8.8 port 53
(coredns) with PID: 3494 sent UDP to 8.8.8.8 port 53
The script will first be compiled. Then, it will start running and printing the data of the UDP packets currently being sent.
We can also redirect the output of the script to a file using this command:
$ stap -v udp_monitor.stp > output_file
The code above adds the output of the script to a file named output_file. This way, we can analyze the output at a later point.
To test the script using ping, we can enter the ping command in a separate terminal while the script is running. We should see something similar to this in our output:
$ (ping) with PID: 16691 sent UDP to 10.199.227.16 port 53
Systemtap is a really powerful tool that has a lot of additional features such as conditions and filters we can add to our script.
4. Conclusion
In this article, we monitored which processes are sending UDP packets from a given machine.
To do this, we explored two different tools: the Linux Audit service and Systemtap. We learned how to use both of them to identify the PID of the processes sending the UDP traffic.
Basically, the Linux Audit service might be easier to use and more beginner-friendly, while Systemtap is usually more powerful and customizable.