1. Overview

In this tutorial, we’ll discuss how to monitor disk I/O activity in the Linux system. It’s an important task to perform while maintaining a system. Essentially, getting data back from the disk costs time. As a result, the disk I/O subsystem is considered the slowest part and can slow down the whole system.

Therefore, it’s important to check disk performance. We need to see if there’s a bottleneck and what causes it, as well as identify which process is waiting for the I/O request to finish. To this end, we’ll learn how to use tools like iostat, iotop, sar, and vmstat to check the disk I/O performance in Linux systems.

2. Install the Tools

Before we begin, we have to install the tools in our Linux system if they aren’t already available. The tools iostat, vmstat, and sar are part of the sysstat package. The iotop tool is part of the iotop package.

To install in Debian, Ubuntu, or any other derivative, we’ll run:

$ sudo apt-get install sysstat
$ sudo apt-get install iotop

For RHLE, CentOS, and Fedora, we’ll run:

$ sudo dnf install sysstat
$ sudo dnf install iotop

3. Report Disk I/O Statistics

To start, it’s interesting to get an overview of disk I/O activity. In this case, the iostat command is handy and easy to understand. It stands for input/output statistics. It reports information about the CPU and disk device utilization.

At this time, we’ll only focus on disk I/O activities; therefore, we’ll use the -d option:

$ iostat -d 
Linux 5.13.12-100.fc33.x86_64 (dhcppc5)     08/30/2021     _x86_64_    (4 CPU)

Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
dm-0              4.84        48.24        35.44         0.00   15064145   11067448          0
dm-1              1.00         1.41         2.60         0.00     440364     812036          0
dm-2              0.18         5.59         0.01         0.00    1747005       2788          0
dm-3              0.08         8.02         0.58         0.00    2503710     180532          0
dm-4              0.00         0.00         0.00         0.00       1272          0          0
dm-5              0.08         8.02         0.58         0.00    2503421     181956          0
sda               3.14        63.31        38.14         0.00   19772796   11911070          0
scd0              0.00         0.00         0.00         0.00          1          0          0
zram0             7.74         9.36        21.58         0.00    2924188    6738944          0

To repeat the iostat command, we can add an interval in second:

$ iostat -d 2

As we can see, iostat displays information about all device activities present in the system. We can also add the -p option to display statistics for block devices and all of their partitions:

$ iostat -d -p sda
Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
sda               4.11        78.09        47.02         0.00   77566501   46701318          0
sda1              0.00         0.01         0.00         0.00       8432          6          0
sda2              4.10        78.08        47.02         0.00   77550214   46701280          0
sda3              0.00         0.01         0.00         0.00       5168         32          0

By default, the iostat command displays information about all block devices and partitions present in the system. If we only want to print information about the disks or partitions that are really active, skipping devices with zero metrics, we can add the -z option:

$ iostat -d -z 2
Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
dm-0              0.50         2.00         0.00         0.00          4          0          0
sda               0.50         2.00         0.00         0.00          4          0          0
zram0             2.50        10.00         0.00         0.00         20          0          0
Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
dm-0             22.50         0.00       152.00         0.00          0        304          0
dm-3              0.50        64.00         0.00         0.00        128          0          0
dm-5              0.50        64.00         0.00         0.00        128          0          0
sda              16.00        64.00       148.00         0.00        128        296          0
zram0             0.50         2.00         0.00         0.00          4          0          0

4. Identify the Process Behind the Bottleneck

We previously discussed how to check I/O activities, but sometimes that isn’t enough. For instance, it’s important to identify which process or thread is causing heavy I/O activities. The iotop command can help us do just that. It’s an interactive tool that provides real-time disk activities by process or thread. It’s important to note that iotop requires root privileges or NET_ADMIN capability.

Also, we have to check if the following kernel options are enabled:

$ egrep '(CONFIG_VM_EVENT_COUNTERS|TASK_IO_ACCOUNTING|CONFIG_TASKSTATS|TASK_DELAY_ACCT)' /boot/config-$(uname -r)
CONFIG_TASKSTATS=y
CONFIG_TASK_DELAY_ACCT=y
CONFIG_TASK_IO_ACCOUNTING=y
CONFIG_VM_EVENT_COUNTERS=y

By default, the iotop command displays a view of disk I/O by each process or thread:

$ sudo iotop
Total DISK READ :      18.27 K/s | Total DISK WRITE :       0.00 B/s
0.00 B DISK READ:      18.27 K/s | Actual DISK WRITE:      14.61 K/s
    TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND                                                                                                
    509 be/3 root        0.00 B/s    0.00 B/s  0.00 %  3.91 % [jbd2/dm-0-8]
 247269 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.21 % [kworker/0:2-events]
   2507 be/4 nwd        18.27 K/s    0.00 B/s  3.57 %  0.00 % gnome-terminal-server
      1 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % systemd --switched-root --system --deserialize 30
      2 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kthreadd]
      3 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcu_gp]
      4 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcu_par_gp]
      6 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kworker/0:0H-events_highpri]
      9 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [mm_percpu_wq]
     10 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcu_tasks_kthre]
     11 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcu_tasks_rude_]
     12 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcu_tasks_trace]
     13 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/0]
     14 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcu_sched]

We can use the -o option to show only the processes or threads actually performing I/O activity:

$ sudo iotop -o 
Total DISK READ :    1864.38 K/s | Total DISK WRITE :       0.00 B/s
10.34l DISK READ:    1875.65 K/s | Actual DISK WRITE:       0.00 B/s
    TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND                                                                                                
 245778 be/4 nwd       608.93 K/s    0.00 B/s 62.08 % 34.04 % firefox -contentproc -childID 3 -isForBrowser -prefs~46 -appdir /usr/lib64/firefox/browser 245562 true tab
   1822 be/4 nwd       105.25 K/s    0.00 B/s  0.00 % 14.13 % gnome-shell
   2507 be/4 nwd       139.08 K/s    0.00 B/s  0.00 %  1.48 % gnome-terminal-server
 245562 be/4 nwd       206.74 K/s    0.00 B/s 93.98 %  0.00 % firefox
 246053ble_ponwd ]      18.79 K/s    0.00 B/s 24.22 %  0.00 % firefox -contentproc -childID 5 -isForBrowser -prefs~sr/lib64/firefox/browser 245562 true tab [DOM Worker]
 247924 be/4 nwd       127.80 K/s    0.00 B/s 44.84 %  0.00 % vlc --started-from-file /home/nwd/Music/Mozart Allegro assai.mp4
 247956 be/4 nwd       657.79 K/s    0.00 B/s 38.89 %  0.00 % dd if=/dev/zero of=/tmp/test.img bs=1G oflag=dsync

We can add the -P option to display the list of processes without threads:

$sudo iotop -oP

5. Generate Disk I/O Statistics Over a Period of Time

We previously learned how to monitor disk activities in real-time, but there are situations where we need to collect historical data for an entire week or more. The goal is to collect enough data for analysis or to plan future evolution. The simplest way to do this is by using the sar command*.*

By default, the sar command monitors all the system resources. In our case, we’re only interested in disk activities. Let’s use the -b option to report details about the disk activities:

$ sar -b 1 
...
04:10:00 PM       tps      rtps      wtps      dtps   bread/s   bwrtn/s   bdscd/s
04:10:01 PM     12.00      0.00     12.00      0.00      0.00    176.00      0.00
04:10:02 PM     35.00     31.00      4.00      0.00   1272.00     80.00      0.00
04:10:03 PM      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:10:04 PM     73.00      0.00     73.00      0.00      0.00    648.00      0.00
04:10:05 PM      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:10:06 PM     22.00      0.00     22.00      0.00      0.00    192.00      0.00
04:10:07 PM      1.00      0.00      1.00      0.00      0.00      8.00      0.00
04:10:08 PM      4.00      0.00      4.00      0.00      0.00    112.00      0.00

We can report activity for each block device using the -d option. We can also identify devices easily by using the -p option:

$ sar -p -d -b 1
04:26:15 PM       DEV       tps     rkB/s     wkB/s     dkB/s   areq-sz    aqu-sz     await     %util
04:26:16 PM       sda      2.00    128.00     52.00      0.00     90.00      0.00      0.50      0.40
04:26:16 PM       sr0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:26:16 PM   lv-root      1.00      0.00     52.00      0.00     52.00      0.00      1.00      0.20
04:26:16 PM   lv-swap      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:26:16 PM     zram0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:26:16 PM   lv-home      1.00    128.00      0.00      0.00    128.00      0.00      1.00      0.20
...

It’s possible to save the output of the sar command in a file with the -o option. Note that the file is in a binary format:

$ sar 2 5 -o /tmp/data_io > /dev/null 2>&1

Now we’ll use the -f option to read the report generated by the sar command saved in the file:

$ sar -f /tmp/data_io

6. Measure Disk I/O Usage With vmstat

Another helpful tool is vmstat, which gives an overall view of what’s happening in the system. For example, we can use the vmstat command to report information about processes, memory, and CPU and show disk I/O activity.

To retrieve disk I/O statistics from the system using vmstat, we invoke:

$ vmstat [-D] [-d] [-p partition] [interval [count]]

We’ll use the -d option to display individual disk statistics:

$ vmstat -d 1
isk- ------------reads------------ ------------writes----------- -----IO------
total merged sectors      ms  total merged sectors      ms    cur    sec
sda   2659118 1531393 166084689 49623955 1725402 4672072 104580917 70770454      0  25194
sr0    24187     28 5886957 1315209      0      0       0       0      0    966
dm-0  2178560      0 114338954 50785360 3433060      0 81458000 79180110      0  21302
dm-1  1605211      0 12845592 24735782 2946339      0 23570712 947720915      0   5866
zram0 2609461      0 20875688   26812 4958733      0 39669864   57039      0    511
dm-2  312586      0 21187074 3814922   2852      0   49288   76928      0    674
dm-3   88452      0 17673344  640177   4957      0  526864  104362      0    537
...

Then we’ll introduce the -p option to obtain detailed performance statistics about a partition:

$ vmstat -p /dev/sda2 1 
sda2            reads      read sectors      writes  requested writes
              2663617         166204812     1725915         104595152
              2663617         166204812     1725915         104595152
              2663617         166204812     1725915         104595152
              2663618         166205068     1725915         104595152
              2663618         166205068     1725917         104595216
...

7. Conclusion

In this article, we learned how to use the iostat, vmstat, and sar commands to check disk I/O performance. We also discussed how to check disk read and write activity by process using the iotop command.