1. Introduction
While debugging or solving an issue with one of our services, typically the first place we want to look at is the logs of our system and services. While we’re doing this, we want to be as specific as possible and avoid sifting through thousands of log entries from different sources.
Most major Linux distributions use systemd for the management of services and system components. Therefore, we need to know how its log manager utility, journalctl, works to be able to check logs effectively on a Linux system.
In this article, we’re going to see how to use the journalctl tool to check, filter, and clean up the logs on a Linux system.
2. Basic journalctl Commands
systemd stores system and service logs in a binary format. This means we can’t use regular text processing tools like cat, tail, grep, sed, or awk directly to read our logs, because our logs aren’t stored as plain text files.
Therefore, we need the journalctl command-line tool to first read and output our logs. Let’s start by looking at a few basic journalctl commands:
2.1. Boots
journalctl can retrieve our logs by a specific boot of our system.
To get a list of boots, let’s run:
$ journalctl --list-boots
-136 6e7ae03aa08a4887a6dc376a30ef1fdb Mon 2020-10-12 00:59:31 +03—Mon 2020-10-12 01:02:06 +03
-135 5e8f616769aa4413b198ef30277bf628 Mon 2020-10-12 01:02:38 +03—Mon 2020-10-12 02:55:47 +03
-134 7b839631c6b54dba8e183a0a91f4fa51 Mon 2020-10-12 02:56:29 +03—Wed 2020-10-14 00:16:38 +03
..
-3 539bf36b695f4d69b3474e1efad9f6e6 Wed 2021-01-27 10:25:02 +03—Wed 2021-01-27 20:46:49 +03
-2 d16424126db24a34b6079b933d0cc09a Wed 2021-01-27 20:47:43 +03—Wed 2021-01-27 23:53:16 +03
-1 d4ca259000374bb9a14e383df551aa6e Thu 2021-01-28 10:15:03 +03—Fri 2021-01-29 20:48:03 +03
0 041a418db54e47fabc646e35da606902 Fri 2021-01-29 20:48:41 +03—Sat 2021-01-30 20:57:10 +03
As we see, the first boot of this system is -136th, and the last boot is 0. Each boot also has a distinct hash value.
To get all the logs for the current boot, let’s run:
$ journalctl -b
..
Jan 29 20:48:43 fcivaner-ThinkPad-T480s systemd[1]: Starting Authorization Manager...
Jan 29 20:48:43 fcivaner-ThinkPad-T480s systemd[1]: Reached target Bluetooth.
Jan 29 20:48:43 fcivaner-ThinkPad-T480s grub-common[1146]: ...done.
Jan 29 20:48:43 fcivaner-ThinkPad-T480s systemd[1]: Started LSB: Record successful boot for GRUB.
Jan 29 20:48:43 fcivaner-ThinkPad-T480s apport[1111]: ...done.
Jan 29 20:48:43 fcivaner-ThinkPad-T480s systemd[1]: Started LSB: automatic crash report generation.
Jan 29 20:48:43 fcivaner-ThinkPad-T480s polkitd[1203]: started daemon version 0.105 using authority implementation `local' version `0.105'
Jan 29 20:48:43 fcivaner-ThinkPad-T480s dbus-daemon[1115]: [system] Successfully activated service 'org.freedesktop.hostname1'
Jan 29 20:48:43 fcivaner-ThinkPad-T480s systemd[1]: Started Hostname Service.
Jan 29 20:48:43 fcivaner-ThinkPad-T480s NetworkManager[1141]: <info> [1611942523.8157] NetworkManager (version 1.10.6) is starting... (for the first time)
..
The log entries we see are from all the services, combined in a single log. We can see that at an early stage when our system is starting up, systemd is starting up services related to Bluetooth, then we see our policy kit (polkitd) starting up, and then our network manager.
To get all the log entries from the previous boot, let’s run:
$ journalctl -b -1
Jan 28 10:15:03 fcivaner-ThinkPad-T480s kernel: microcode: microcode updated early to revision 0xe0, date = 2020-06-17
Jan 28 10:15:03 fcivaner-ThinkPad-T480s kernel: Linux version 5.4.0-64-generic (buildd@lgw01-amd64-030) (gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)) #72~18.04.1-Ubuntu SMP Fri Jan 15 14:06:34 UTC 2021
Jan 28 10:15:03 fcivaner-ThinkPad-T480s kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-5.4.0-64-generic root=UUID=19112d3d-1e88-45c7-be9a-a0d3119a8b18 ro quiet splash vt.handoff=1
Jan 28 10:15:03 fcivaner-ThinkPad-T480s kernel: KERNEL supported cpus:
Jan 28 10:15:03 fcivaner-ThinkPad-T480s kernel: Intel GenuineIntel
Jan 28 10:15:03 fcivaner-ThinkPad-T480s kernel: AMD AuthenticAMD
Jan 28 10:15:03 fcivaner-ThinkPad-T480s kernel: Hygon HygonGenuine
Jan 28 10:15:03 fcivaner-ThinkPad-T480s kernel: Centaur CentaurHauls
Jan 28 10:15:03 fcivaner-ThinkPad-T480s kernel: zhaoxin Shanghai
Jan 28 10:15:03 fcivaner-ThinkPad-T480s kernel: x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'...
...
As we can see from our boot list, our current boot will always have an index of 0, and our last boot will always have an index of -1. We can see the logs for a specific boot by specifying it after the -b argument.
2.2. Filtering by Date or Time
If we want to filter logs by time, we can use –since and –until parameters. Let’s go ahead and try it:
$ journalctl --since="2021-01-30 18:17:16"
Jan 30 18:33:53 fcivaner-ThinkPad-T480s wpa_supplicant[1153]: wlp61s0: WPA: Group rekeying completed with 18:31:bf:c2:98:40 [GTK=CCMP]
Jan 30 18:34:01 fcivaner-ThinkPad-T480s CRON[12094]: pam_unix(cron:session): session opened for user root by (uid=0)
Jan 30 18:34:01 fcivaner-ThinkPad-T480s CRON[12095]: (root) CMD ( test -x /etc/cron.daily/popularity-contest && /etc/cron.daily/popularity-contest --crond)
Jan 30 18:34:01 fcivaner-ThinkPad-T480s CRON[12094]: pam_unix(cron:session): session closed for user root
Jan 30 18:36:58 fcivaner-ThinkPad-T480s kernel: wlp61s0: AP 18:31:bf:c2:98:40 changed bandwidth, new config is 2417 MHz, width 1 (2417/0 MHz)
...
We can also use a relative time parameter to filter quickly:
$ journalctl --since "20 min ago"
...
To learn how the time parsing works, we can refer to the systemd time specification.
3. Narrowing It Down Further
3.1. Units
systemd identifies resources (services, sockets, mounts..) as units. We can get the logs for a specific unit using the -u flag:
$ journalctl -u netcfg
It is also possible to specify a pattern to match one or more units. For example, to see the logs of all the units starting with “systemd-“, we run:
$ journalctl -u systemd-*
If a unit isn’t a system service, but a service we defined as a user, we can check its logs using;
$ journalctl --user-unit my-application
Note that when we use –user-unit, we can only see the logs of the units that are associated with our own user.
3.2. Users and Processes
We can also filter messages for units that belong to a specific user ID or a specific process ID:
$ journalctl _UID=100
$ journalctl _PID=1
3.3. Priority
We may also want to see messages with a certain priority level, such as errors or warnings.
$ journalctl -p err..alert
Here are the codes we can use to filter our messages:
code
priority
0
emerg
1
alert
2
crit
3
err
4
warning
5
notice
6
info
7
debug
Note that a process should output appropriate level prefixes with each log message (such as “<4>” or “
4. Tailing and Following Logs
To follow the logs instead of printing them, we use:
$ journalctl -f
For example, let’s say we want to follow the logs of the service apache. Let’s run:
$ journalctl -f -u apache
To just print the last 10 log messages, use the -n parameter:
$ journalctl -u apache -n 10
5. Disabling the Pager to Get Direct Output
Sometimes, we may want to send our logs to a file, pipe them to another program for processing, or simply use another pager. In this case, we can use the –no-pager option so that journalctl won’t automatically open a pager for us to explore the logs:
$ journalctl --no-pager
To use less as the pager, let’s try:
$ journalctl --no-pager | less -S
6. Vacuuming Logs
Over time, logs on our system can get very large and cause problems. These problems may range from disk space issues to even crashing our system entirely.
Luckily, we can use journalctl to clean up old logs quickly. For example, to clean up our logs to get their disk space usage below 100 MB, we can use:
journalctl --vacuum-size=100M
Or to clean up our logs so that it contains no data older than 2 weeks, we can use:
journalctl --vacuum-time=2weeks
Keep in mind that journald should have previously marked logs as “archived” to be able to vacuum them. It does this only after they become expired by the parameters we define in the journald configuration file (/etc/systemd/journald.conf).
These parameters are:
- SYSTEMKEEPFREE
- RUNTIMEKEEPFREE
- SYSTEMMAXFILESIZE
- RUNTIMEMAXFILESIZE
- MAXRETENTIONSEC
- MAXFILESEC
These parameters determine how much of our logs will be marked as archived. We’re only able to vacuum the logs that are marked as archived, so it’s important for us to adjust how much of our logs we’re going to keep.
When we define these parameters in /etc/systemd/journald.conf, we’ll most likely want to adjust the ones starting with “SYSTEM“. The “RUNTIME” parameters are only applied when our logs are stored in an in-memory filesystem. Usually, in most Linux systems we’ll use, the logs will be stored on a persistent filesystem.
“KEEPFREE” parameters determine how much space journald will try to keep free for other users to use, and “MAXFILESIZE” parameters determine the maximum size that active journal entries can take up on disk. If there’s a conflict, the smallest space between the two is preferred.
To show total disk usage of all journal files, we can use:
journalctl --disk-usage
7. Giving Permission to a User to See System Logs
By default, only the users with specific privileges (root, sudoers, or wheel) can see the system logs. If we don’t have these privileges, we may get an error message:
"No journal files were opened due to insufficient permissions."
While doing system administration, we may want a user without root privileges to be able to see all logs for debugging. Members of the groups systemd-journal, adm, and wheel/sudo can read all journal files. On the other hand, groups adm and wheel/sudo often also have other administrative privileges on the system.
Therefore, it’s a better choice to use systemd-journal as the group that we’ll add our user to.
To permit a user to read the system logs, run:
$ usermod -a -G systemd-journal USER
Where we replace USER with the username we want to give access to.
8. Output Format
We may also want to format our logs differently, perhaps to send them to other systems to store and analyze. We can use the -o parameter to do this:
$ journalctl -b -u docker -o json
As expected, we’ll then see the entries in JSON format:
{
"__CURSOR": "s=188c4f9b9b9148e8b51e1d83acece38c;i=3262ca;b=1b845a1c4af044128eb82da1b49f8a9e;m=14cfbac;t=5bb02ecf82bd3;x=bdd8df9be3959797",
"__REALTIME_TIMESTAMP": "1612996123569107",
"__MONOTONIC_TIMESTAMP": "21822380",
"_BOOT_ID": "1b845a1c4af044128eb82da1b49f8a9e",
"PRIORITY": "6",
"_UID": "0",
"_GID": "0",
"_SELINUX_CONTEXT": "unconfined\n",
"_SYSTEMD_SLICE": "system.slice",
"_MACHINE_ID": "ed152e13dffc4a0aa8ab86f0b781cd23",
"_HOSTNAME": "fcivaner-ThinkPad-T480s",
"SYSLOG_FACILITY": "3",
"_CAP_EFFECTIVE": "3fffffffff",
"_TRANSPORT": "stdout",
"SYSLOG_IDENTIFIER": "dockerd",
"_COMM": "dockerd",
"_EXE": "/usr/bin/dockerd",
"_CMDLINE": "/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock",
"_SYSTEMD_CGROUP": "/system.slice/docker.service",
"_SYSTEMD_UNIT": "docker.service",
"_STREAM_ID": "61ec716173cb4e52bd940795991b3463",
"_PID": "2002",
"_SYSTEMD_INVOCATION_ID": "5f3e883264d347c0aec150e82c6b79e5",
"MESSAGE": "time=\"2021-02-11T01:28:43.568571629+03:00\" level=info msg=\"Docker daemon\" commit=46229ca graphdriver(s)=overlay2 version=20.10.3"
},
...
9. Conclusion
In this tutorial, we’ve learned how to use the journalctl tool to view, filter, and clean up our logs, and how to configure our system so a user can see the system logs using journalctl.