1. Overview
When working with Linux servers, we may encounter the “Too many open files” error. In this article, we’ll go over what this error means and how we can fix it.
2. What Is a File Descriptor?
From regular data to network sockets, everything is a file in Linux! A file descriptor is a non-negative integer identifier for an open file in Linux. Each process has a table of open file descriptors where a new entry is appended upon opening a new file.
For example, what happens when we cat a file? It opens the file passed as an argument via the open() system call and gets assigned a file descriptor for it. Then, it interacts with the file through the file descriptor – in this case, just for showing its contents – and finally, closes it via the close() system call.
A process has three file descriptors open by default, denoted by 0 for stdin, 1 for stdout, and 2 for stderr.
3. Checking Open File Descriptors
We can check the number of file descriptors used by various agents to determine where our resources are being used.
3.1. Global Usage
If we want to check the total number of file descriptors open on the system, we can use an awk one-liner to find this in the first field of the /proc/sys/fs/file-nr file:
$ awk '{print $1}' /proc/sys/fs/file-nr
2944
3.2. Per-Process Usage
We can use the lsof command to check the file descriptor usage of a process. Let’s take the caddy web server as an example:
$ sudo lsof -p $(pidof caddy)
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
caddy 135 caddy cwd DIR 254,1 4096 1023029 /etc/sv/caddy
caddy 135 caddy rtd DIR 254,1 4096 2 /
caddy 135 caddy txt REG 254,1 33595392 1542819 /usr/bin/caddy
caddy 135 caddy 0u CHR 5,1 0t0 12 /dev/console
caddy 135 caddy 1u CHR 5,1 0t0 12 /dev/console
caddy 135 caddy 2w FIFO 0,9 0t0 119 pipe
caddy 135 caddy 3u a_inode 0,10 0 6 [eventpoll:2,4,6,7,8,9,10,11,13,15,16,17,18,19,20,21,22,23,24,25,27,28,29,30,31,32,33,34]
caddy 135 caddy 4r FIFO 0,9 0t0 171 pipe
caddy 135 caddy 5w FIFO 0,9 0t0 171 pipe
caddy 135 caddy 6u IPv4 234 0t0 TCP localhost.localdomain:2019 (LISTEN)
caddy 135 caddy 7u IPv6 240 0t0 TCP *:443 (LISTEN)
caddy 135 caddy 8u IPv6 241 0t0 TCP *:80 (LISTEN)
caddy 135 caddy 9u IPv6 3819034 0t0 TCP 66.46.134.82.baeldung.com:443->google.com:53222 (ESTABLISHED)
We can see the exact file that a given file descriptor belongs to under the NAME column, while its type is located under the TYPE column. Looking at the entries with IPv6 as their type, we can conclude that even network sockets take up file descriptors, just like regular files.
4. File Descriptor Limits
A process has certain limits with regard to the number of file descriptors it can have open at a time. One is a soft limit, which can be changed by any unprivileged user and can never exceed the hard limit. An unprivileged user can lower the hard limit but not raise it again, whereas a privileged user such as root can raise and lower it as required.
4.1. Per-Session Limit
We use the ulimit command with the -Sn flag to check the soft limit, and the -Hn flag to check the hard limit for the current session:
$ ulimit -Sn
1024
$ ulimit -Hn
4096
4.2. Per-Process Limit
We can check a process’s limits, given its PID, via the procfs filesystem:
$ pid=31540
$ grep "Max open files" /proc/$pid/limits
Max open files 1024 4096 files
The second and third fields correspond to the soft and hard limits, respectively.
4.3. Global Limit
There’s a system-wide limit to the total number of file descriptors that can be open by all the processes combined. This limit is stored in the /proc/sys/fs/file-max file:
$ cat /proc/sys/fs/file-max
1634185
5. Increasing File Descriptor Limits
Now that we have a good understanding of the idea behind the “Too many open files” error, let’s go over various ways to solve it. We can verify these changes with the commands mentioned in the previous section. We use 500000 to refer to the desired limit under this section’s examples.
5.1. Temporarily (Per-Session)
Let’s try to configure the limit for our current session with the ulimit -n command:
$ ulimit -n 4096
$ ulimit -n
4096
$ ulimit -n 8192
sh: error setting limit: Operation not permitted
We cannot set the limit above 4096 as that is the hard limit in this case. And as noted above, only a privileged user can change the hard limit.
5.2. Per-User
We can change the soft and hard limits globally for all processes by appending a few lines to the /etc/security/limits.conf file and re-logging in:
* hard nofile 500000
* soft nofile 500000
root hard nofile 500000
root soft nofile 500000
In the first field, we specify the user that the limit affects. We set the limit for all users on the system with the * glob.
Additionally, we might need to append a line to /etc/pam.d/common-session on some systems:
session required pam_limits.so
5.3. Per-Service
On systemd-based distributions, we use the systemctl command to configure the limits for specific services. Taking apache as an example, let’s create a filelimit.conf file for it with tee:
$ sudo mkdir -p /etc/systemd/system/apache.service.d/
$ sudo tee /etc/systemd/system/apache.service.d/filelimit.conf <<EOF
[Service]
LimitNOFILE=500000
EOF
Now, we simply reload the configuration and restart the service:
$ sudo systemctl daemon-reload
$ sudo systemctl restart apache.service
5.4. Globally
Earlier, we checked the system-wide aggregate limit in the /proc/sys/fs/file-max file. We can configure it with sysctl by appending a line to the /etc/sysctl.conf file:
fs.file-max = 500000
Finally, we need to run the sysctl -p command to reload the configuration file.
6. Conclusion
In this article, we learned about the working of Linux File Descriptors, including various ways of checking and configuring their limits depending on our needs. Along the way, we saw how to check the maximum number of open file descriptors globally, per-session, and per-process, as well as ways to configure the limits at the user, session, process, and service levels.