How to Log Events and Maintain Compliance with journald, the New Linux System Event Log– Part 1: Configuring the Daemon

Cybersecurity laws and other regulations clearly exist for good reasons, and there are serious penalties involved if you fail to meet them. The new version of PCI DSS, the Payment Card Industry Data Security Standard, requires going beyond showing that data can be secure, you must show that it will be secure through established procedures that are carefully followed. That means automated collection of log data followed by periodic analysis.

You have to reliably collect it in the first place before you can process and analyze it, and that brings us to the relatively new journal mechanism in recent releases of Linux.

This new Linux journal collects (or logs) the types of events you’re accustomed to getting through the Syslog or RSyslog, which you also have.  But it can collect more than (R)Syslog can.  It includes a daemon – journald – analogous to the syslogd or rsyslog daemon.  Unlike the old style of saving data in a file, the journal exists at least in running memory and, if you ask for it, in a persistent file.  You don’t read the journal directly, you use the journalctl command to control and query it.  And if that isn’t enough new features and names, the service that starts the journal is actually journald-systemd!

StenographyI have recently written about systemd, here and here, describing its entirely new philosophy that includes parallelization of the boot process, shutdown, and all other system state changes.

One of the earliest daemons started is systemd-journald. It collects messages much like the Syslog (or Rsyslog, or syslog-ng) daemon you may already be familiar with. Messages can come from the kernel or from user-space processes. A logging daemon “sees” all of them.

With Syslog variants, including Rsyslog and syslog-ng, the daemon has to be told what to keep and where to keep it. The default configuration is likely what you want, but definitely check to make sure!

The older versions collected their data in text files, one line per message, a format ready for easy processing with grep, awk, and increasingly complex tools including Perl and commercial applications like Splunk and ArcSight.  Many of these topics are covered in our Introduction to Perl Programming class.

Benefits of journald-systemd

Like much of the rest of systemd, I think you will first notice just how very different the journal is in comparison to the traditional Syslog family of tools. For example, there is no simple file you can process and examine with traditional tools. Depending on how you have it configured, there may be a binary file that you must examine with the journalctl command, or there may be no file at all as the journal is kept in RAM.

But then you discover many things that you can easily accomplish with the journal that would require intricate coding with traditional log files. For example, the ability to extract only log events from an arbitrary period of time — “All of yesterday”, or “From 08:12 until 09:57 this morning”, or “From the previous boot session, from when the system booted the time before this session until its shutdown.”

Another benefit is that the journal is started very early, before the file system where logs are stored is remounted in read-write mode. So it can capture some events that Syslog simply cannot.

It’s important to realize that the journal provides tools with which you can achieve compliance, but it’s no magical fix. You have to be careful to use the tools correctly.

Much like using systemctl to control and query systemd services and system states, we do most everything journal-related with the journalctl command. It also automatically truncates its otherwise very wide output lines, and uses a pager so you can easily interact with it.

Go ahead and start a terminal emulator, stretch it horizontally as wide as you can, and become root with the su - command.

You need to become root because anyone can run journalctl but you can only see your own events unless you are root or a member of the systemd-journal group. Run journalctl — like last week, I’ll show you the commands but leave it up to you to view their rather large output.

# journalctl

Notice that it starts by telling you the period covered by the logs. That leads us to the issue of log storage. You configure the journal in /etc/journald.conf, see the manual page for journald.conf for all the details. A line in that file starting Storage= specifies how to store journal entries. Explicitly setting this to persistent causes the daemon to create and use a hierarchy under /var/log/journal/. For regulatory compliance, your auditors will probably want to see persistent storage on disk but this is not the default setting on many distributions including Red Hat Enterprise Linux.

If you specify volatile the journal data will be stored in memory only and be lost at reboot time. Using auto, which has the same effect as not specifying anything, means that if a directory /var/log/journal happens to exist then the journal will be stored persistently, but if that directory isn’t there it will be stored in memory only.

One risk with logging is that you can run out of space on that file system. The journal daemon automatically enforces size limits on what is stored persistently. You can tune that behavior in the configuration file.

Another configuration change you may want to make would be to forward log messages to a traditional Syslog daemon. The journal daemon can’t send and receive messages across the network, at least not yet, so careful centralized log collection and analysis needs to connect the journal daemon to Rsyslog in order to transmit the log messages over a TLS-protected connection. You learn how to do that in the Linux server administration course.

Type to search

Do you mean "" ?

Sorry, no results were found for your query.

Please check your spelling and try your search again.