1. Overview

With the advancements in hardware and cloud technologies, a single Linux machine can host various applications and VMs. Therefore, it’s pivotal that we manage system resources like CPU usage.

We often need to use commands like top or htop to view the available/used CPU for a machine. However, if we want to view such stats for a specific process, these commands are not as straightforward.

In this tutorial, we’ll look at a few options we can use to gather per-core CPU usage of a process.

2. The top Command

Generally speaking, the top command is the most popular choice for monitoring system resources. Simple syntax and the detailed snapshot of system resources makes top a common choice for identifying the CPU usage of the machine:

$ top
top - 14:11:06 up 3 days, 19:12,  0 users,  load average: 1.07, 1.18, 0.91
Tasks:   4 total,   1 running,   3 sleeping,   0 stopped,   0 zombie
%Cpu(s):  4.5 us,  4.7 sy,  0.0 ni, 90.2 id,  0.5 wa,  0.0 hi,  0.2 si,  0.0 st
KiB Mem :  2015684 total,   114416 free,  1100304 used,   800964 buff/cache
KiB Swap:  1048572 total,  1010160 free,    38412 used.   813428 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
    1 root      20   0   32632   1504   1412 S   0.0  0.1   0:00.05 nginx
    6 nginx     20   0   33112    672    180 S   0.0  0.0   0:00.00 nginx
  682 root      20   0   18132   3128   2688 S   0.0  0.2   0:00.02 bash
  687 root      20   0   41060   3104   2600 R   0.0  0.2   0:00.00 top

Notably, top displays the process-wise CPU usage together with the overall CPU usage. Furthermore, it also displays the utilization for all the users in the machine — for example, root and nginx from our output above.

However, we can see that this output doesn’t display per-core CPU usage. It only shows the overall used or idle CPU.

If we want to know the CPU usage of each core, we can press “1” while the top command is active:

$ top
top - 14:27:13 up 3 days, 19:28,  0 users,  load average: 0.29, 0.46, 0.62
Tasks:   4 total,   1 running,   3 sleeping,   0 stopped,   0 zombie
%Cpu0  :  5.9 us,  4.9 sy,  0.0 ni, 87.8 id,  1.4 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu1  :  4.5 us,  3.8 sy,  0.0 ni, 89.9 id,  1.4 wa,  0.0 hi,  0.3 si,  0.0 st
KiB Mem :  2015684 total,   112124 free,  1100472 used,   803088 buff/cache
KiB Swap:  1048572 total,  1010160 free,    38412 used.   813268 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
    1 root      20   0   32632   1504   1412 S   0.0  0.1   0:00.05 nginx
    6 nginx     20   0   33112    672    180 S   0.0  0.0   0:00.00 nginx
  682 root      20   0   18132   3128   2688 S   0.0  0.2   0:00.02 bash
  688 root      20   0   41060   3212   2708 R   0.0  0.2   0:00.49 top

Now, we can see that CPU0 utilizes 5.9%, and CPU1 utilizes 4.5% of the CPU on this two-core machine. However, as shown above, we still can’t figure out the per-core CPU usage of a specific process.

To know the core utilization, we must identify the core on which the kernel has scheduled the process threads. We can also use the taskset command to add affinity so that the kernel always schedules the process on the configured core.

3. The sar Command

The sar command is part of the sysstat package in Linux, which provides various system utilities. Additionally*,* the sysstat package contains a system data collector that runs as a cron job and collects all system statistics and stores them in data files. We can then view these stats with the help of the sar command.

In most machines, sysstat comes pre-installed. However, we can also install it manually using Linux package installers like Yum and Apt.

Similar to the top command, the sar command also displays the overall CPU usage by default:

$ sar -u 1 3
Linux 4.19.76-linuxkit (nginx-deployment-574b87c764-fwgw8)      08/31/20        _x86_64_        (2 CPU)

02:28:58        CPU     %user     %nice   %system   %iowait    %steal     %idle
02:28:59        all      3.66      0.00      3.66      0.52      0.00     92.15
02:29:00        all      4.74      0.00      5.79      1.58      0.00     87.89
02:29:01        all      3.66      0.00      4.71      0.00      0.00     91.62
Average:        all      4.02      0.00      4.72      0.70      0.00     90.56

Here, the arguments 1 and 3 denote that sar should print the CPU information after every one second for a maximum of three times. In order to get the per-core CPU usage, we’ll use the -P argument with the sar command:

$ sar -P ALL 1 3
Linux 4.19.76-linuxkit (nginx-deployment-574b87c764-fwgw8)      08/31/20        _x86_64_        (2 CPU)

02:36:34        CPU     %user     %nice   %system   %iowait    %steal     %idle
02:36:35        all      3.11      0.00      3.11      0.00      0.00     93.78
02:36:35          0      3.12      0.00      2.08      0.00      0.00     94.79
02:36:35          1      3.06      0.00      4.08      1.02      0.00     91.84

02:36:35        CPU     %user     %nice   %system   %iowait    %steal     %idle
02:36:36        all      3.59      0.00      3.08      0.51      0.00     92.82
02:36:36          0      3.09      0.00      3.09      0.00      0.00     93.81
02:36:36          1      4.12      0.00      2.06      0.00      0.00     93.81

02:36:36        CPU     %user     %nice   %system   %iowait    %steal     %idle
02:36:37        all      2.58      0.00      3.09      0.52      0.00     93.81
02:36:37          0      2.04      0.00      4.08      1.02      0.00     92.86
02:36:37          1      3.06      0.00      3.06      1.02      0.00     92.86

Average:        CPU     %user     %nice   %system   %iowait    %steal     %idle
Average:        all      3.09      0.00      3.09      0.34      0.00     93.47
Average:          0      2.75      0.00      3.09      0.34      0.00     93.81
Average:          1      3.41      0.00      3.07      0.68      0.00     92.83

Here, ALL denotes that sar should display the CPU utilization for all cores. We can further filter this output to a specific core by replacing ALL with the core id like 0 or 1.

Again, taskset can be used to set core affinity to a process so that we can filter our sar output to a particular core.

4. The ps Command

Generally, we use the ps command to list the processes running on a Linux machine. We can get information like process id, threads, run command, process owner, etc. from the output of this command:

$ ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 Sep01 ?        00:00:00 nginx: master process nginx -g daemon off;
nginx        6     1  0 Sep01 ?        00:00:00 nginx: worker process
root         7     0  0 13:24 pts/0    00:00:00 bash
root       676     7  0 14:02 pts/0    00:00:00 ps -ef

Similar to the above statistics, we can also see CPU% for each process with the -o argument. This argument allows us to define a custom output format from the ps command:

$ ps -eo uid,pid,tid,class,rtprio,ni,pri,psr,pcpu,comm
  UID   PID   TID CLS RTPRIO  NI PRI PSR %CPU COMMAND
    0     1     1 TS       -   0  19   0  0.0 nginx
  101     6     6 TS       -   0  19   0  0.0 nginx
    0     7     7 TS       -   0  19   0  0.0 bash
    0   687   687 TS       -   0  19   0  0.0 ps

Here, %CPU is the CPU usage of each thread with thread id %TID of the process. Since each thread is scheduled on a different core, we can say that this denotes the per-core CPU usage.

Although ps is the quickest way to get per-core utilization, it’s not the most reliable. It’s important to point out here that ps is a snapshot tool rather than a monitoring tool like top. Therefore, the CPU usage shown here is the average CPU time consumed by the process since it was started. 

On the other hand, the output of the top command is the actual CPU consumed by the process at any given time. The top command monitors the CPU consumption and hence reports the most accurate CPU consumption of a process.

5. The mpstat Command

Similar to the sar command, the mpstat command is also a part of the sysstat package. This command reports the per-core CPU usage for all processes in the machine:

$ mpstat -P ALL 3
Linux 4.19.76-linuxkit (nginx-deployment-574b87c764-fwgw8)      09/03/20        _x86_64_        (2 CPU)

12:39:34     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
12:39:37     all    5.91    0.00    5.74    0.52    0.00    0.35    0.00    0.00    0.00   87.48
12:39:37       0    6.25    0.00    5.56    0.35    0.00    0.00    0.00    0.00    0.00   87.85
12:39:37       1    5.56    0.00    6.25    0.69    0.00    0.35    0.00    0.00    0.00   87.15

12:39:37     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
12:39:40     all    3.44    0.00    4.81    0.52    0.00    0.00    0.00    0.00    0.00   91.24
12:39:40       0    3.42    0.00    5.14    0.68    0.00    0.34    0.00    0.00    0.00   90.41
12:39:40       1    3.79    0.00    4.48    0.00    0.00    0.00    0.00    0.00    0.00   91.72
^C

Average:     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
Average:     all    4.67    0.00    5.27    0.52    0.00    0.17    0.00    0.00    0.00   89.37
Average:       0    4.83    0.00    5.34    0.52    0.00    0.17    0.00    0.00    0.00   89.14
Average:       1    4.67    0.00    5.36    0.35    0.00    0.17    0.00    0.00    0.00   89.45

The command we used will display the overall CPU usage along with per-core usage after every 3 seconds. 

This output is similar to the top and the sar command. As we’ve discussed previously, if we know the cores on which our process is running, we can easily know the CPU usage for it.

6. The dstat Command

The dstat command is part of the dstat package in Linux. Similar to other commands discussed here, dstat is a statistical command that displays system resources like CPU usage.

One major advantage that dstat has over other commands is that it allows us to input the CPU core id for which we need the CPU usage:

$ dstat -C 0,1
You did not select any stats, using -cdngy by default.
-------cpu0-usage--------------cpu1-usage------ -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq:usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw
  5   5  89   1   0   0:  5   5  89   1   0   0|  29k   64k|   0     0 |  49B  496B| 676  5163
  5   3  90   2   0   0:  5   2  93   0   0   0|   0    72k|   0     0 |   0     0 | 685  5292
  3   4  93   0   0   0:  3   5  92   0   0   0|   0    12k|   0     0 |   0     0 | 855  5589

The command shown above outputs the CPU usage for core 0 and core 1. On Linux machines, we can find the core id from the cpuinfo file:

$ cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 142
model name      : Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz
stepping        : 10
microcode       : 0xffffffff
cpu MHz         : 2112.000
cache size      : 8192 KB
.
.
.
processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 142
model name      : Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz
stepping        : 10
microcode       : 0xffffffff
cpu MHz         : 2112.000
cache size      : 8192 KB
.
.
.

7. Conclusion

So, with all things considered, we can say that there’s no shortcut command to get the per-core CPU utilization for a process in Linux. And, in most cases, we need to control the cores on which a process can run to get this information. We briefly discussed the taskset command, which allows us to add CPU core affinities so that a process is scheduled on that core.

Later, we also discussed a few commands in Linux that can give us the per-core CPU utilization. Therefore, if we know which core our process is scheduled for, we can easily get the CPU usage for that core using these commands.