1. Introduction
Modern software often updates views automatically based on data changes:
- head and tail
- git (Global Information Tracker), to follow file changes for commits
- vscode (Visual Studio Code)
This convenience is afforded by different systems.
In this tutorial, we talk about the inotify system and how it functions. After that, we explore ways of checking its current users. Finally, we discuss some common errors linked with inotify.
We tested the code in this tutorial on Debian 11 (Bullseye) with GNU Bash 5.1.4. It is POSIX-compliant and should work in any such environment.
2. Filesystem Event Notifications
We’ve discussed inodes as a key in native Linux filesystems. In fact, their change is linked with many file events.
Based on this, the inotify subsystem’s main functions are to monitor and report on events in the filesystem. Whether with the inotifywait and inotifywatch tools, or system calls from other software, we can easily keep track of file operations.
For example, we can use a command line to simply wait for any event on a given file:
$ inotifywait /file.ext && echo 'Event.'
Setting up watches.
Watches established.
Consequently, if /file.ext is now part of any operation, we’ll get a notification. But what if we wanted to permanently do that for many objects at once?
3. Monitoring Multiple Files and Directory Trees
There are times when we need more general monitoring. One way is to add the –recursive (-r) and –monitor (-m) switches to inotifywait. They ensure directory tree processing and constant monitoring, respectively.
Additionally, we can use inotifywatch to listen for events linked with a given part of the filesystem:
$ inotifywatch /
Establishing watches...
Finished establishing watches, now collecting statistics.
Since a separate watch is needed for each filesystem object, Linux has an upper limit on its maximum number. In fact, we can check what it is in a given system:
$ cat /proc/sys/fs/inotify/max_user_watches
524288
Here, the system allows 524288 (2^19) watches. Conveniently, we can see who uses them at any moment.
4. Checking inotify Users
By combining several Linux commands, we can find out the processes that currently use watches:
$ inotifywatch / &
Establishing watches...
Finished establishing watches, now collecting statistics.
[1] 666
$ find /proc/*/fd -lname anon_inode:inotify |
cut -d/ -f3 |
xargs -I '{}' -- ps --no-headers -o '%p %U %c' -p '{}' |
uniq -c |
sort -nr
1 6660 root inotifywatch
The number of inotify file descriptors is not the same as the number of watches. Because of this, there is a script, which contains a better version of the above command line along with other features.
Additionally, we can use strace (System Trace) to get a list of all watches:
$ strace --follow-forks --trace='inotify_add_watch' inotifywait --quiet /file.ext
inotify_add_watch(3, "/file.ext",
IN_ACCESS|IN_MODIFY|IN_ATTRIB|IN_CLOSE_WRITE|IN_CLOSE_NOWRITE|IN_OPEN|IN_MOVED_FROM|IN_MOVED_TO|IN_CREATE|IN_DELETE|IN_DELETE_SELF|IN_MOVE_SELF) = 1
Finally, there is also the lsof (List Open Files) tool, which also lists inotify users:
$ tail -f /file.ext &
[1] 666
$ lsof | grep inotify
tail 666 root 4r a_inode 0,11 0 15387 inotify
Even with all the ways to use and handle watches, issues are not rare.
5. Common inotify Errors
There are pitfalls when using the inotify subsystem. Let’s discuss some of them.
5.1. Permissions
Indeed, as with other filesystem operations, we need to have at least some control over the target objects. In other words, we can’t monitor without proper rights:
$ inotifywatch /root
Establishing watches...
Failed to watch /root: Permission denied
To fix this, we need to be able to at least read the target object. Here, we add the necessary right via chmod (Change Mode):
$ chmod +r /root
After these changes, we can add watches.
5.2. inotify Watch Limit
Consider that a recursive inotifywatch on / would take up as many watches as there are files. Indeed, doing that on a filesystem with more files than the watch limit would produce an error:
$ cat /proc/sys/fs/inotify/max_user_watches
524288<
$ df --inodes /
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sdb 16777216 666000 16111216 4% /
$ inotifywatch --recursive /
Establishing watches...
Failed to watch /; upper limit on inotify watches reached!
Please increase the amount of inotify watches allowed per user via '/proc/sys/fs/inotify/max_user_watches'.
Of course, the solution is in the error text. However, there is more than one way to apply it:
$ max_user_watches=CUSTOM_MAX_USER_WATCHES_VALUE
$ sysctl fs.inotify.max_user_watches=${max_user_watches}
$ echo fs.inotify.max_user_watches=${max_user_watches} >> /etc/sysctl.conf
$ sysctl --load
While the first command applies it per session, the second survives a reboot. That’s because it adds a line to /etc/sysctl.conf. However, if the line is already there, we should change it per our needs. We can avoid a restart with the last command, which rereads and applies the configuration file.
5.3. Main Memory (RAM) Issues
Because each watch is a structure, available memory is also a bottleneck for using inotify. In fact, a watch can take up to 1KB of space. This means a million watches could result in 1GB of extra RAM usage.
Sometimes, this can lead to strange errors:
- inotify_add_watch failed: No space left on device (/file.ext)
- cannot watch ‘/file.ext’: No space left on device
Freeing up memory seems like the way to go. However, there are a couple of problems with this approach.
Firstly, inotify uses kernel memory, which can’t be swapped. Secondly, programs can throw such errors even when just running out of watches.
6. Summary
In this article, we discussed the inotify Linux subsystem, used by many applications.
In conclusion, knowing about watches allows us to consider not only the inner workings of software products but also understand errors we see during their use.