1. Introduction
In this tutorial, we’ll learn how to find which apps or processes are using a specific kernel module. Kernel modules are pieces of code that Linux can load into the kernel to add features like device drivers, file systems, or system calls.
Throughout this guide, we’ll use Ubuntu 22.04 “Jammy Jellyfish” with the 5.15.0-35-generic kernel. However, the steps should work on other modern Linux distributions as well.
2. Identifying Functions to Trace for Each Kernel Module
First, we need to find the functions linked to a module. One way to do this is to use the /proc/kallsyms file, which holds the kernel’s symbol table. By filtering for the module name in this file, we find the functions it contains.
For example, to get the functions linked to the e1000 module:
$ cat /proc/kallsyms | grep e1000
ffffffffc039e89b t e1000_setup_rctl.cold [e1000]
ffffffffc038bcf0 t e1000_configure_rx [e1000]
ffffffffc038c880 t e1000_clean_rx_irq [e1000]
ffffffffc038d890 t e1000_clean_jumbo_rx_irq [e1000]
ffffffffc038eff0 t e1000_alloc_rx_buffers [e1000]
ffffffffc039e8b0 t e1000_configure_rx.cold [e1000]
ffffffffc038bf40 t e1000_tbi_should_accept [e1000]
ffffffffc039e8c5 t e1000_tbi_should_accept.cold [e1000]
# ... ...
Although we could trace all functions connected to a specific module, doing so may generate too much data. Thus, it’s better to concentrate on a few essential functions.
3. Standard Tools for Gathering Information on Kernel Modules
3.1. lsmod
lsmod shows the status of modules presently loaded in the kernel. It reads the information from the /proc/modules file.
To list the modules installed and loaded on the system, we can simply run the following:
$ lsmod
Module Size Used by
# ... Other modules ...
rc_core 65536 1 cec
psmouse 176128 0
drm 606208 6 drm_kms_helper,drm_vram_helper,vboxvideo,drm_ttm_helper,ttm
e1000 155648 0
mptbase 106496 2 mptspi,mptscsih
The output shows the modules’ name, size, and number of instances of each specific module that are currently in use. The comma-separated list after the number shows what’s using the module, which could be a device name, a file system identifier, or the name of another module.
3.2. modinfo
modinfo is a command that extracts information about a module from its associated .ko (kernel object) file.
For example, to display information about the e1000 module, we can run the following:
$ modinfo e1000
filename: /lib/modules/5.15.0-35-generic/kernel/drivers/net/ethernet/intel/e1000/e1000.ko
license: GPL v2
description: Intel(R) PRO/1000 Network Driver
author: Intel Corporation, <[email protected]>
srcversion: B4D2D3AB3E4A089C43DAE56
alias: pci:v00008086d00002E6Esv*sd*bc*sc*i*
# ... ...
This output provides information about the module, such as its filename, version, license, description, author, and parameters.
4. Using bpftrace to Trace Kernel Module Usage
bpftrace is a tracing tool that uses the Linux kernel’s eBPF (extended Berkeley Packet Filter) functionality to trace events in the kernel. Moreover, it has a powerful scripting language allowing us to write custom scripts to trace events, filter data, and perform other operations.
In this example, we opt for the e1000 module and the e1000_clean_rx_irq function because:
- The e1000 module is a common network driver
- The e1000_clean_rx_irq is a function responsible for handling received packets. That makes it a suitable candidate for observing process interactions
Let’s now proceed with the necessary steps.
First, we need to install bpftrace:
$ sudo apt install bpftrace
Then, run this bpftrace command, filtering for ping calls:
$ sudo bpftrace -e 'kprobe:e1000_clean_rx_irq { printf("%s(%d) called e1000_clean_rx_irq\n", comm, pid); }' | grep ping
Now, in another terminal, let’s run ping:
$ ping 8.8.8.8 -c 10
The bpftrace output shows the ping process:
ping(49398) called e1000_clean_rx_irq
We have successfully traced the e1000_clean_rx_irq function and found that it was called by the ping process. This information can be useful for debugging or performance analysis.
5. Using trace-cmd to Trace Kernel Module Usage
trace-cmd is a command-line tool for tracing kernel events. It uses the ftrace facility to trace events in the kernel. It’s a powerful alternative to bpftrace to find processes accessing a specific kernel module in our system.
Let’s quickly go through another simple example. However, this time, we’ll use the ext4 module and the ext4_file_write_iter function, responsible for writing data to ext4 file systems.
As ext4 is the default file system format for the Ubuntu operating system, it’s a viable candidate.
Before proceeding, it’s important to note that the ext4 module is not present in the output of lsmod. This is because the ext4 module is built into the kernel and is not a separate module.
First, install trace-cmd:
$ sudo apt install trace-cmd
Next, enable the function tracer and specify the function to trace:
$ sudo trace-cmd record -p function -l ext4_file_write_iter
plugin 'function'
Hit Ctrl^C to stop recording
In another terminal, create a file to trigger the ext4_file_write_iter function:
$ echo "Hello World" > ~/test.txt
Now, let’s stop the tracing by pressing Ctrl+c in the terminal running trace-cmd. The trace data will be saved to a file named trace.dat
To view the trace data, run the following:
$ trace-cmd report | grep ext4_file_write_iter
CPU 1 is empty
cpus=2
bash-50632 [000] 131048.988738: function: ext4_file_write_iter
Lastly, the output shows that the bash process called the ext4_file_write_iter function.
6. Conclusion
In this tutorial, we’ve covered tools and methods to help us find out what processes or apps use a specific kernel module. We’ve discussed common tools like lsmod and modinfo, and we also saw how to use bpftrace to trace kernel module usage. Finally, we’ve explored using trace-cmd as an alternative tool.
There are many other tools and methods to trace kernel module usage. Some of them are strace, perf, and systemtap. Each of these tools has its advantages, disadvantages, and degree of complexity.
By learning how to trace kernel module usage and analyze process-module interactions, we gain valuable information for debugging, optimizing performance, or resolving conflicts in our Linux system.