1. Overview

Linux, by its design, aims to use all of the available physical memory as efficiently as possible. But, at times, system resource limitations can cause abrupt behavior on the server-side. Usually, these limitations trigger high CPU and high memory usage. In any case, we could really avoid such abrupt behavior by observing the memory utilization of a process.

In this tutorial, we’ll learn to implement a few tips and tricks using some well-known Linux commands, to help us recognize the peak memory usage of a process.

2. Traditional Commands to Monitor Memory

For the most part, commands like top**/htop/atop give us the processes overview. In specific cases, they may also be used to monitor a particular process. Here, we’re focusing on checking a process to identify its peak memory utilization.

We can start by investigating the top result to see the process overview. This gives an idea regarding what all processes are using.

Let’s say we need to zero in on a specific process, and we know its process id (PID):

$ top -p 7
top - 10:25:53 up 19 min,  0 users,  load average: 0.52, 0.58, 0.59
Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
%Cpu(s): 16.0 us,  6.2 sy,  0.0 ni, 77.1 id,  0.0 wa,  0.7 hi,  0.0 si,  0.0 st
MiB Mem :   3961.1 total,    665.1 free,   3072.0 used,    224.0 buff/cache
MiB Swap:  12288.0 total,  11806.3 free,    481.7 used.    758.5 avail Mem
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
    7 user1     20   0   18076   3632   3528 S   0.0   0.1   0:00.19 bash

Now we’ll look at bash usage:

$ top | grep bash
    7 user1     20   0   18076   3632   3528 S   0.0   0.1   0:00.19 bash
   73 root      20   0   18076   3604   3540 S   0.0   0.1   0:00.17 bash

htop is similar to top, but shows more data about a process. It has the command column, which is convenient to recognize the process path.

atop is again a command like top and htop. Its advantage is the very useful feature of recording the output in a file.

Consider an issue that happens once and again at a particular time window. To keep a record, we can schedule a cron job to write the output to a file, so it becomes feasible to play back later. We’ll use atop -w if we need to record the output to a file:

$ atop -w filename

And we’ll use atop -r if we need to playback the output from that file:

$ atop -r filename
  PID      SYSCPU       USRCPU        VGROW        RGROW       RUID           EUID           ST       EXC        THR       S       CPUNR        CPU       CMD        1/1   73       0.17s        0.03s       481.1G        3604K       root           root           N-         -          1       S           0         0%       bash
    7       0.10s        0.09s       591.0G        3632K       user1          user1          N-         -          1       S           0         0%       bash
    1       0.15s        0.00s       376.9G         316K       root           root           N-         -          2       S           0         0%       init
   71       0.04s        0.00s       716.4G        2908K       root           root           N-         -          1       S           0         0%       sudo
   72       0.03s        0.00s       820.0G        2092K       root           root           N-         -          1       S           0         0%       su
  109       0.01s        0.00s         1.0T        2136K       root           root           N-         -          1       R           0         0%       atop
    6       0.00s        0.01s       376.9G         224K       root           root           N-         -          1       S           0         0%       init

These three commands can be the best tools for a continuous investigation. For our situation, we can identify the peak memory usage of a process before it arrives at the set limit.

3. grep One-Liner

The /proc virtual filesystem is a directory containing the hierarchy of files that represent the current state of a Linux kernel. It also includes information on any currently running processes.

Here’s a one-liner that determines the peak memory usage of one such process having the process id (PID) 113:

$ grep ^VmPeak /proc/113/status 
VmPeak: 2252 kB

We can also look for “VmHWM: Peak resident set size” to measure RAM usage. VmPeak is the maximum total memory usage, including virtual memory, while VmHWM is the peak RAM usage.

As we know, /proc is a virtual filesystem, so reading from its files is not the same as reading from a normal filesystem. The information about a process is removed from /proc much faster than if it was a real filesystem (dirty cache flushing involved here).

With this in mind, imagine that we need to read the next line of a process that isn’t buffered yet. In this case, the information about it may have already been removed. We may not need information about a process that no longer exists. The solution is to either account for file loss or buffer the entire file and then parse it.

4. GNU time

Let’s understand the GNU time in the given context by going through a few examples.

Suppose we want to know the peak memory usage of the ‘top‘ process. In this case, it’s the “Maximum resident set size” that tells us so:

$ /usr/bin/time -v top | grep "Maximum resident set size"
Maximum resident set size (kbytes): 2252

GNU time supports the format option. Apart from memory usage, GNU time with the %P option provides unrelated statistics (%CPU), which depends on the scheduler and is therefore quite variable:

$ /usr/bin/time -f "%P %M" top
2% 2248

In bash, we need to specify the full path, such as /usr/bin/time, because the bash built-in time keyword doesn’t support the -f option:

$ /usr/bin/time -f '%M' top 
2248

We can also create an alias or adjust the environment to use GNU time for average and maximum memory information:

alias time="$(which time) -f '\t%E real,\t%U user,\t%S sys,\t%K amem,\t%M mmem'"
export TIME="$(which time) -f '\t%E real,\t%U user,\t%S sys,\t%K amem,\t%M mmem'"

This information stored in memory segments represents the average total (data+stack+text) memory use (K), and the maximum resident set size (M) of the process.

4.1. Arguments and Considerations

Let’s do a time command fact-check by discussing some of its known issues.

The first issue is that time may be broken for memory reporting on some Linux systems. When we determine “Maximum resident set size” using /usr/bin/time -v ls, we can see that most of the time, it returns a 0. It always returns 0 because time derives most information from the wait3(2) system call.

Systems that don’t have a wait3(2) call, use the time(2) system call instead. However, it provides much less information than wait3(2). So, such systems report the majority of the resources as 0.

Alternatively, trying a more CPU-intensive command may do the trick.

The second issue is that upon calling time -v, the output “bash: -v: command not found” means bash intercepts time to use its own built-in time function. Using /bin/time -v solves the problem.

Thirdly, GNU time has a bug. It reports 4x the actual memory usage. time 1.7-24 on Ubuntu 14.4 and Fedora’s time package from version 1.7-3 incorporates a fix to the memory reporting.

4.2. Built-in time vs. GNU time

As we noted above, in bash, we need to specify the full path to GNU time, for example, /usr/bin/time. Alternatively, we can use command time -l.

It’s important to note that the command is not a placeholder. The command time is different than just time. The command time -l invokes the shell to call a binary called time instead of the built-in function.

5. Valgrind One-Liner

Valgrind is a framework that provides instrumentation to user-space binaries. It ships with several tools that can be used to profile and analyze program performance.

Massif, one of the Valgrind tools, measures the heap memory used by a specific program. A simple valgrind massif one-liner to depict peak memory usage of the ‘top‘ process will look like:

$ valgrind --tool=massif --pages-as-heap=yes --massif-out-file=massif.out top; grep mem_heap_B massif.out | sed -e 's/mem_heap_B=\(.*\)/\1/' | sort -g | tail -n 1
==746== Massif, a heap profiler
==746== Copyright (C) 2003-2017, and GNU GPL'd, by Nicholas Nethercote
==746== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==746== Command: top
==746==
==746== error calling PR_SET_PTRACER, vgdb might block
top - 21:47:03 up 23 min,  0 users,  load average: 0.52, 0.58, 0.59
Tasks:   8 total,   1 running,   6 sleeping,   1 stopped,   0 zombie
%Cpu(s):  2.8 us,  1.9 sy,  0.0 ni, 95.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :   3961.1 total,    408.3 free,   3328.8 used,    224.0 buff/cache
MiB Swap:  12288.0 total,  12014.8 free,    273.2 used.    501.7 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
    1 root      20   0    8940    308    268 S   0.0   0.0   0:00.20 init
    6 root      20   0    8940    220    180 S   0.0   0.0   0:00.01 init
    7 user1     20   0   18080   3456   3348 S   0.0   0.1   0:00.27 bash
   72 root      20   0   18924   2900   2800 S   0.0   0.1   0:00.10 sudo
   73 root      20   0   18048   2080   2056 S   0.0   0.1   0:00.07 su
   74 root      20   0   18076   3608   3516 S   0.0   0.1   0:00.41 bash
  366 root      20   0   18444   1784   1324 T   0.0   0.0   0:00.01 top
  746 root      20   0   70528  26284   3072 R   0.0   0.6   0:01.24 massif-amd64-li

==746==
15691776

The output 15691776 is the peak memory usage of the ‘top‘ process.

By default, Massif measures only heap memory. However, if we wish to measure all the memory used by a program, we can use —pages-as-heap=yes.

Massif outputs profiling data to a massif.out file. The ms_print tool graphs this profiling data to show memory consumption over the execution of a program. It also shows detailed information about the sites responsible for allocation at points of peak memory allocation. We can graph the data from the massif.out file with the command:

ms_print massif.out

6. Other Contemporary Tools: Heaptrack and Busybox

Heaptrack is a KDE tool that has both GUI and text interfaces. It provides peak memory usage as flame graphs. It’s faster as it does less checking than Valgrind.

We can determine peak memory by tracking the amount of PSS in /proc/[pid]/smaps or use pmap. We can also attach heaptrack to an already running process:

heaptrack --pid $(pid of <your application>)

heaptrack output is written to /tmp/heaptrack.APP.PID.gz.

Busybox combines tiny versions of many common UNIX utilities into a single small executable. It provides minimalist replacements for most of the utilities we usually find in GNU Coreutils, until-Linux, and others.

Busybox is pre-installed in most Linux distros, including Debian and Ubuntu. It can be used in place of commands not present in many modern distributions. Likewise, to know peak memory usage, we can use the busybox time implementation with the -v argument:

$ /usr/bin/time busybox time -v uname -r | grep "Maximum resident set size"
Maximum resident set size (kbytes): 1792

Its output is similar to GNU time output.

7. Conclusion

In this article, we discussed commands and tools to identify peak memory usage of a process.

These commands help us visualize the process’ memory utilization in real-time so we can take necessary action. Additionally, they can assist us with gauging activities to decrease the application’s memory use.