1. Introduction
systemd is an init system and service manager in Linux. It handles various tasks, including booting the system, starting/stopping services, and managing logs.
More importantly, systemd uses a centralized logging mechanism called journald. The journald daemon collects messages from the kernel, system processes, and applications, storing them in a binary format for efficient access.
In this tutorial, we’ll explore how to capture system logs with only systemd-journald. The commands in this tutorial work on Debian-based distributions (like Ubuntu), but the concepts apply to all Linux distributions.
2. Understanding syslog and systemd-journald
On Unix-based operating systems such as Linux, keeping a record of events and errors is important for troubleshooting and security purposes. To achieve this, syslog, a standard protocol, is used for message logging. syslog enables programs to send event and error messages to a central location for storage.
Traditionally, the syslogd daemon received and managed these messages. However, recent distributions such as Ubuntu 20.04 have upgraded to rsyslogd, an advanced logging facility that builds upon and extends the functionalities of syslogd.
On the other hand, systemd-journald* also stores all system logs in a central location, typically */var/log/journal. These logs are stored in binary format and are lightweight, consuming little storage space yet storing more data.
Furthermore, the binary log file format provides a layer of security as binary logging formats are less prone to manipulation. Additionally, it’s possible to detect if the files have been modified.
To interact with these logs, we use the journalctl command-line tool. It enables us to view, filter, and manage entries within the journal. We can even export logs in different formats for further analysis.
However, since both syslogd and systemd-journald capture logs, this leads to duplicate logging. In the later sections, we’ll explore how to make systemd-journald the primary logging facility to avoid this potential redundancy.
3. Configuring systemd-journald as the Primary Logging System
In this section, we’ll configure systemd-journald as the primary logging system for our Linux machine and disable the existing logging system as well.
3.1. Check for systemd-journald
First, let’s check if systemd-journald is already running and enabled on the machine:
$ systemctl status systemd-journald
● systemd-journald.service - Journal Service
Loaded: loaded (/lib/systemd/system/systemd-journald.service; static)
Active: active (running) since Mon 2024-04-08 06:43:46 UTC; 10min ago
TriggeredBy: ● systemd-journald.socket
● systemd-journald-audit.socket
● systemd-journald-dev-log.socket
Docs: man:systemd-journald.service(8)
man:journald.conf(5)
Main PID: 356 (systemd-journal)
Status: "Processing requests..."
Tasks: 1 (limit: 498)
Memory: 4.6M
CPU: 325ms
CGroup: /system.slice/systemd-journald.service
└─356 /lib/systemd/systemd-journald
...
In the output, the indicator of whether the service is running is the line starting with Active: followed by the service’s state. If it says active (running), then systemd-journald is running. However, if it’s not running, we need to enable and start it:
$ systemctl enable systemd-journald
$ systemctl start systemd-journald
Again, check the status of systemd-journald to confirm it’s running:
$ systemctl status systemd-journald
Once it’s confirmed that systemd-journald is running on the Linux machine, we can configure it as the primary logging facility.
3.2. Stop Log Forwarding in journald.conf
The systemd-journald service stores its configuration file by default at /etc/systemd/journald.conf. Let’s look into the file with the cat command:
$ sudo cat /etc/systemd/journald.conf
# Use 'systemd-analyze cat-config systemd/journald.conf' to display the full config.
#
# See journald.conf(5) for details.
[Journal]
#Storage=auto
#Compress=yes
#Seal=yes
#SplitMode=uid
#SyncIntervalSec=5m
#RateLimitIntervalSec=30s
#RateLimitBurst=10000
#SystemMaxUse=
#SystemKeepFree=
#SystemMaxFileSize=
#SystemMaxFiles=100
#RuntimeMaxUse=
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#RuntimeMaxFiles=100
#MaxRetentionSec=
#MaxFileSec=1month
#ForwardToSyslog=yes
#ForwardToKMsg=no
#ForwardToConsole=no
#ForwardToWall=yes
#TTYPath=/dev/console
#MaxLevelStore=debug
#MaxLevelSyslog=debug
#MaxLevelKMsg=notice
#MaxLevelConsole=info
#MaxLevelWall=emerg
#LineMax=48K
#ReadKMsg=yes
#Audit=no
As we can see from the output, the configuration displays commented parameters. This indicates that systemd recognizes the parameter values as default. Therefore, any modification requires uncommenting the desired parameter and restarting the systemd-journald service to save the changes.
The ForwardToSyslog parameter in journald.conf is set to yes by default. To stop forwarding logs to syslog, we set the ForwardToSyslog parameter to no.
Let’s edit journald.conf in a text editor such as vim and make our change:
$ sudo vim /etc/systemd/journald.conf
We need to uncomment the ForwardToSyslog parameter and change the value to no:
ForwardToSyslog=no
Let’s save and exit the file. Finally, we need to restart the systemd-journald service to enact the changes:
$ systemctl restart systemd-journald
This change stops forwarding logs captured by the systemd-journald daemon to the syslogd daemon (in this case, rsyslogd). Hence, this will prevent duplicate logs.
3.3. Stop syslog Logging in rsyslog.conf
The default rsyslog configuration is in /etc/rsyslog.d/50-defualt.conf. Again, let’s look into this file with cat:
$ sudo cat /etc/rsyslog.d/50-default.conf
# Default rules for rsyslog.
#
# For more information see rsyslog.conf(5) and /etc/rsyslog.conf
#
# First some standard log files. Log by facility.
#
auth,authpriv.* /var/log/auth.log
*.*;auth,authpriv.none -/var/log/syslog
#cron.* /var/log/cron.log
#daemon.* -/var/log/daemon.log
kern.* -/var/log/kern.log
#lpr.* -/var/log/lpr.log
mail.* -/var/log/mail.log
#user.* -/var/log/user.log
#
# Logging for the mail system. Split it up so that
# it is easy to write scripts to parse these files.
#
#mail.info -/var/log/mail.info
#mail.warn -/var/log/mail.warn
mail.err /var/log/mail.err
...
As we can see from the output, the configuration file defines how rsyslog categorizes and stores various system logs.
To stop sending messages to /var/log/syslog, let’s edit this file in vim:
$ sudo vim /etc/rsylog.d/50-default.conf
Then, we need to comment out the following line and save the file:
# *.*;auth,authpriv.none -/var/log/syslog
After saving the file, we can restart rsyslog to update the changes:
$ systemctl restart rsyslog
This change in rsyslog configuration stops sending log messages to /var/log/syslog.
3.4. Disable rsyslog.service and syslog.socket
Another option for stopping duplicate logs and using systemd-jounald as the only logging facility is to stop the rsyslog service completely. However, disabling rsyslog.service alone won’t be enough; we also have to disable syslog.socket.
syslog.socket is the systemd socket that listens for incoming log messages sent over a network using the traditional syslog protocol. Disabling it ensures that even if rsyslog were somehow started, it wouldn’t be able to receive messages sent over the network via syslog.
Let’s disable rsyslog.service and syslog.socket:
$ sudo systemctl disable --now syslog.socket rsyslog.service
Therefore, disabling both rsyslog.service and syslog.socket guarantees that syslog functionality is completely stopped, leaving systemd-journald as the primary logging facility.
4. Conclusion
In this article, we’ve learned how to disable the existing logging system and configure systemd-journald as the primary logging facility on the Linux machine. Additionally, we looked at the syslog protocol that handles message logging on the Linux machine.
We also covered the traditional syslogd and the benefits of using systemd-journald.