Earlier this week I told you how to set up the journal daemon. Now that it has had time to gather data, let’s see how to easily extract meaningful reports from the Linux system log files.
First, let’s see the complete journal data. Look at the first line to see how far back the journal goes, and then look through at least a few pages. Remember to do this in a terminal window that you have stretched horizontally, and do this as
root or, to be more cautious, a user who is a member of group
Wow, that’s a lot! If you only know how to display the complete contents of the journal, your problem is that you have too much information. Of course you could pipe that output through
awk or other utilities, but a great thing about the journal is that you can ask it to only show you certain things.
The journal entries are terse. The journal tools include a catalog of explanations, you can ask for those to be added as available. You can add this option to any of the following examples:
# journalctl --catalog
So far we’re just exploring out of curiosity, and the journal data is overwhelmingly complete and verbose. Let’s narrow our search to investigate a specific service. This is a more typical use of the journal:
# journalctl --unit=sshd
There are just the SSH events — when the service has started and stopped, and the authentication successes and failures.
We can also make selections based on time. One way of doing this is in terms of the booted session. Session #1 is from the very first entry in the journal until the first shutdown, #2 is from when it booted back up for the second time until shutdown, and so on. Negative numbers are backwards from the end, session -0 is the current or latest one, -1 was the previous session lasting until the most recent shutdown, and so on. If you specify some boot with
-b but you don’t say which one, it defaults to the current booted session:
# journalctl -b
You can combine these selectors. What about the SSH events during the previous booted session?
# journalctl --unit=sshd -b -1
You can also use time references. Maybe you suspect that your server underwent an SSH password guessing attack this morning:
# journalctl --unit=sshd --since=06:00:00 --until=12:00:00
Or maybe it was some time between the last morning of February through the afternoon of the following day:
# journalctl --unit=sshd --since="2015-02-28 06:00:00" --until="2015-03-01 18:00:00"
Or has the sensor daemon detected any temperature or fan speed problems since midnight?
# journalctl --since=today --unit=sensord
Wow! That’s much easier than figuring out how to extract those lines with
Another thing that I have found handy is to have a terminal emulator with a small font up in the corner of the display, so I can easily notice if something is causing rapid logged events. I’ve done that with this command:
# tail -f /var/log/messages
Lots of us find that handy, so they built the equivalent trick into
# journalctl -f
The Linux server administration course teaches you how to set up many Linux services and storage. The journal lets you monitor them.