1. Overview

In this tutorial, we’ll get a deeper understanding of monitoring device events in Linux. First, we’ll overview the key terms like kernel, udev, and udevadm, followed by hands-on exercises to solidify our knowledge.

2. Basics of Device Events in Linux

In Linux, devices are files in the /dev/ directory. Let’s run a command to see a partial listing of those particular files:

$ ls -lrt /dev/
total 0
drwxrwxrwt  2 root  root          40 Jul 23 11:24 mqueue
drwxr-xr-x  2 root  root          60 Jul 23 11:24 mapper
drwxr-xr-x  2 root  root          60 Jul 23 11:24 dma_heap
drwxr-xr-x  2 root  root          60 Jul 23 11:24 vfio
drwxr-xr-x  2 root  root          60 Jul 23 11:24 net
...

But how are these files created? To answer this, we can insert a flash drive into our system. As a result, a file for the device is either dynamically created or becomes accessible in the /dev/ directory.

In detail, this is made possible by a userspace program called udev, a dynamic device manager for the Linux kernel. Whenever a device is added or removed, the kernel sends a notification of the change to udev.

So, udev listens for these uevents from the kernel. For every event, systemd-udevd (the udev daemon) runs specific instructions defined for handling each device. In fact, the /etc/udev/rules.d/ and /lib/udev/rules.d/ directories define this set of rules.

Below are the contents of /etc/udev/rules.d/:

$ ls -l /etc/udev/rules.d/
total 84
-rw-r--r-- 1 root root 10302 Jul 11 22:37 70-snap.firefox.rules
-rw-r--r-- 1 root root  1389 Jul 11 22:37 70-snap.snapd-desktop-integration.rules
-rw-r--r-- 1 root root 63312 Jul 11 22:37 70-snap.snapd.rules
-rw-r--r-- 1 root root  3121 Jul 11 22:37 70-snap.snap-store.rules

The above output displays the custom rules for udev.

Next, let’s see a partial listing of /lib/udev/rules.d/:

$ ls -l /lib/udev/rules.d/
total 624
-rw-r--r-- 1 root root   991 Mac 25 12:53 39-usbmuxd.rules
-rw-r--r-- 1 root root   998 Jan 20  2019 40-usb-media-players.rules
-rw-r--r-- 1 root root 42861 Sep  6  2021 40-usb_modeswitch.rules
-rw-r--r-- 1 root root   655 Apr  7 22:28 40-vm-hotadd.rules
-rw-r--r-- 1 root root   165 Okt 26  2021 50-apport.rules
...

The output above shows the default rules for udev.

3. Monitor Device Events with udevadm

In this section, we’ll monitor the handling of devices. Importantly, we can manipulate the behavior of the udev daemon using the udevadm tool.

3.1. Using the udevadm monitor Command

The udevadm tool monitors a device from when it’s plugged into the system to when it’s removed. So, to display this activity, we can use the udevadm monitor command:

$ udevadm monitor
monitor will print the received events for:
UDEV - the event which udev sends out after rule processing
KERNEL - the kernel uevent

We can see that the udevadm tool has now started monitoring device events. So now, let’s see what happens when we plug a disk into our system:

$ udevadm monitor
monitor will print the received events for:
UDEV - the event which udev sends out after rule processing
KERNEL - the kernel uevent

KERNEL[40495.431179] change   /devices/pci0000:00/0000:00:1f.2/ata2/host1/target1:0:0/1:0:0:0/block/sr0 (block)
UDEV  [40501.736535] change   /devices/pci0000:00/0000:00:1f.2/ata2/host1/target1:0:0/1:0:0:0/block/sr0 (block)
KERNEL[40503.777729] change   /devices/pci0000:00/0000:00:1f.2/ata2/host1/target1:0:0/1:0:0:0/block/sr0 (block)
UDEV  [40506.864856] change   /devices/pci0000:00/0000:00:1f.2/ata2/host1/target1:0:0/1:0:0:0/block/sr0 (block)

We can see some activity. In particular, the device named sr0 at /block/sr0 is the disk we just inserted. First, the kernel sends a notification to udev for a change in that device. Next, udev updates the device’s state.

3.2. Displaying Device Properties

Next, let’s take a look at the device properties by examining the udev database. For this, we’ll need the info command with a path (-p):

$ udevadm info -a -p /block/sr0

Udevadm info starts with the device specified by the devpath and then
walks up the chain of parent devices. It prints for every device
found, all possible attributes in the udev rules key format.
A rule to match, can be composed by the attributes of the device
and the attributes from one single parent device.

  looking at device '/devices/pci0000:00/0000:00:1f.2/ata2/host1/target1:0:0/1:0:0:0/block/sr0':
    KERNEL=="sr0"
    SUBSYSTEM=="block"
    DRIVER==""
    ATTR{alignment_offset}=="0"
    ATTR{capability}=="109"

From the output, we get to know the device attributes in a key-and-value format due to the -a flag. If we look at the attribute SUBSYSTEM, we can see that its value is block, meaning a block device.

4. Monitor Device Events with evtest

evtest is a tool that keeps track of an input device by showing generated events.

First, let’s install the tool:

# apt-get install evtest

Before we continue, it’s important to note that we’ll need to switch to a virtual terminal. But why do that? Well, when a device is active, a process takes it over. This hinders evtest from displaying events. To emphasize, switching to a virtual terminal breaks this hold on the device.

Next, let’s look at the input devices file in the /proc pseudo-system:

$ cat /proc/bus/input/devices
I: Bus=0011 Vendor=0002 Product=0007 Version=01b1
N: Name="SynPS/2 Synaptics TouchPad"
P: Phys=isa0060/serio1/input0
S: Sysfs=/devices/platform/i8042/serio1/input/input5
U: Uniq=
H: Handlers=mouse0 event4 
B: PROP=9
B: EV=b
B: KEY=6420 30000 0 0 0 0
B: ABS=260800011000003
...

The output shows a selected device file which we’ll analyze. In particular, we’ll focus on the /dev/input/event4 file (from the Handlers section above), responsible for mouse input:

# evtest /dev/input/event4
Input driver version is 1.0.1
Input device ID: bus 0x11 vendor 0x2 product 0x7 version 0x1b1
Input device name: "SynPS/2 Synaptics TouchPad"
Supported events:
  Event type 0 (EV_SYN)
  Event type 1 (EV_KEY)
    Event code 272 (BTN_LEFT)
    Event code 273 (BTN_RIGHT)
    Event code 325 (BTN_TOOL_FINGER)
    Event code 330 (BTN_TOUCH)
    Event code 333 (BTN_TOOL_DOUBLETAP)
    Event code 334 (BTN_TOOL_TRIPLETAP)
...

evtest* shows data received from the *kernel. The above snippet is just a part of the long evtest output. It shows basic device data, as well as generated events.

5. Conclusion

In this article, we saw that the udevadm and evtest tools are instrumental when working with device events in the Linux system. Furthermore, we also reviewed important key topics like kernel and userspace udev components. With this knowledge, we can keep track of device events with ease.