1. Overview
In this tutorial, we’ll be looking at the problem of the df command reporting. Specifically, we’ll learn the reasons for the df command giving incorrect disk usage information.
2. The df Command Incorrectly Saying Filesystem Is Full
The df command is a Linux tool that reports disk usage of the file systems on the system. Let’s say we have a partition mounted on the /partition2 directory. We can check the physical disk usage by running df -h /partition2:
$ sudo df -h /partition2
Filesystem Size Used Avail Use% Mounted on
/dev/sdb1 2.0G 24K 1.9G 1% /partition2
The output tells us, out of all the available capacity, how much disk space on that particular filesystem has been consumed and how much is left. From our example, we see that the /dev/sbd1 filesystem has 2GB of space in total, and 24KB has been consumed.
Although a handy tool for checking disk usage in the system, there are some common inconsistencies with the reporting from the df command. Firstly, it can sometimes show a 100% value under the Use% column despite the filesystem still having leftover capacity. For example:
$ Filesystem Size Used Avail Use% Mounted on
/dev/sdb1 2.0G 1.4G 0 100% /partition2
Separately, it sometimes correctly displays the disk usage information, but the creation of files leads to a “No space left on device” error:
$ df -h /partition2
Filesystem Size Used Avail Use% Mounted on
/dev/sdb1 2.0G 517M 1.4G 27% /partition2
$ echo "Hello" > /partition2/file.txt
bash: /partition2/file.txt: No space left on device
These inconsistencies are usually due to one of the following reasons, which we’ll explore in the subsequent sections.
3. Processes Holding Onto Deleted Files
One possible reason for the incorrect reporting is that some processes on the system are still holding up the deleted files. For example, an nginx process is still holding onto a deleted log file. The solution to this issue is to simply kill or restart the process to make it release the file.
Firstly, we can get a list of deleted files that are still being referenced by running processes by running the command lsof +L1:
$ lsof +L1
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
gnome-she 2021 bob 32r REG 8,5 32768 0 2240031 /home/bob/.local/share/gvfs-metadata/root-a2e424ab.log (deleted)
snap-stor 2330 bob 17w REG 8,5 36864 0 2228516 /home/bob/snap/snap-store/common/.cache/appstream/appcache-OGZRZ1.mdb (deleted)
gnome-ter 2856 bob 20u REG 8,5 393216 0 1180433 /tmp/#1180433 (deleted)
less 3214 bob 4r REG 8,17 500000000 0 13 /partition2/file1.txt (deleted)
By default, the lsof command reports all the currently opened files in the system. When we pass the +L1 option, we’re instructing the lsof command to show opened files with less than 1 link. In other words, we want to report currently opened files that are deleted.
As we can see, we have a 500MB file1.txt on the partition2 directory held up by the less command. We can then proceed to reclaim the space by terminating the process. Once that’s done, we should be able to see that the df command is reporting the correct disk usage number:
$ df -h /partition2
Filesystem Size Used Avail Use% Mounted on
/dev/sdb1 2.0G 1.4G 428M 77% /partition2
4. Running out of Inodes
In Linux, each file and directory have an index node (commonly known as an inode) that stores the metadata. Concretely, the inode stores information such as owner, permission bits, timestamps, and location of the file data on the disk. Additionally, there’s a maximum number of inodes we can have on each file system.
Therefore, when we run out of inodes’ capacity, we’ll get the “No space on device” error despite the df command reporting free space available. To check for the current inodes usage, we can run the df command with the -i flag:
$ df -i /partition2
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sdb1 131072 131072 0 100% /partition2
Here it’s showing that we have fully used up all the available inodes on our /dev/sdb1 file system. This is a common problem for a system that contains a large number of small files. To fix this issue, we can archive several large files into a single file. This will save up the inode count as the archive consumes only a single inode.
5. Root User Disk Space Reserves
It’s also worth mentioning that Linux filesystems such as ext3 and ext4 reserve 5% of the total disk space on a filesystem for the root user. This is necessary to ensure the critical function of the system continues to run when the disk is filling up. If the discrepancy we’re observing is roughly 5%, there’s a good chance this is the cause.
If the total disk space for a filesystem is very large, it might make sense to resize the reserves to a lower percentage so that we don’t reserve too much absolute space. To do that, we’ll first need to install the e2fsprogs program using our package manager:
$ sudo apt-get install -y e2fsprogs
Then, we can first check the reserved amount using the dumpe2fs command that comes in the same package:
$ sudo dumpe2fs /dev/sdb1 | grep -i reserved
Reserved block count: 26201
Reserved GDT blocks: 255
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
Reserved GDT blocks at 2-256
Reserved GDT blocks at 32770-33024
Reserved GDT blocks at 98306-98560
Reserved GDT blocks at 163842-164096
Reserved GDT blocks at 229378-229632
Reserved GDT blocks at 294914-295168
To change the percentage of the reserve to 1%, we run the tune2fs command followed by the -m 1 option:
$ sudo tune2fs -m 1 /dev/sdb1
tune2fs 1.45.5 (07-Jan-2020)
Setting reserved blocks percentage to 1% (5240 blocks)
We can then check that the block count of the reserve is now 5240 blocks:
$ sudo dumpe2fs /dev/sdb1 | grep -i reserved
dumpe2fs 1.45.5 (07-Jan-2020)
Reserved block count: 5240
Reserved GDT blocks: 255
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
Reserved GDT blocks at 2-256
Reserved GDT blocks at 32770-33024
Reserved GDT blocks at 98306-98560
Reserved GDT blocks at 163842-164096
Reserved GDT blocks at 229378-229632
Reserved GDT blocks at 294914-295168
6. Conclusion
In this tutorial, we’ve looked at the issue of the df command incorrectly reporting full disk. Then, we learned that processes that are holding up deleted files might cause the issue. Furthermore, we’ve also seen how running out of inodes will cause df to report a full disk. Finally, we’ve also learned that filesystems in Linux generally reserve 5% of total disk space for the root user which might cause the discrepancy observed.