1. Overview
The Linux operating system, and many applications that run on it, do a lot of logging. These logs are invaluable for monitoring and troubleshooting your system.
What you’ll learn
- Viewing logs with a simple GUI tool
- Basic command-line commands for working with log files
What you’ll need
- Ubuntu Desktop or Server
- Very basic command-line knowledge (
cd,ls, etc.)
Originally authored by Ivan Fonseca.
How will you use this tutorial?
-
Only read through it
Read it and complete the exercises
What is your current level of experience?
-
Novice
Intermediate
Proficient
2. Log files locations
There are many different log files that all serve different purposes. When trying to find a log about something, you should start by identifying the most relevant file. Below is a list of common log file locations.
System logs
System logs deal with exactly that — the Ubuntu system — as opposed to extra applications added by the user. These logs may contain information about authorizations, system daemons and system messages.
Authorization log
Location: /var/log/auth.log
Keeps track of authorization systems, such as password prompts, the sudo command and remote logins.
Daemon Log
Location: /var/log/daemon.log
Daemons are programs that run in the background, usually without user interaction. For example, display server, SSH sessions, printing services, bluetooth, and more.
Debug log
Location: /var/log/debug
Provides debugging information from the Ubuntu system and applications.
Kernel log
Location: /var/log/kern.log
Logs from the Linux kernel.
System log
Location: /var/log/syslog
Contains more information about your system. If you can’t find anything in the other logs, it’s probably here.
Application logs
Some applications also create logs in /var/log. Below are some examples.
Apache logs
Location: /var/log/apache2/ (subdirectory)
Apache creates several log files in the /var/log/apache2/ subdirectory. The access.log file records all requests made to the server to access files. error.log records all errors thrown by the server.
X11 server logs
Location: /var/log/Xorg.0.log
The X11 server creates a seperate log file for each of your displays. Display numbers start at zero, so your first display (display 0) will log to Xorg.0.log. The next display (display 1) would log to Xorg.1.log, and so on.
Non-human-readable logs
Not all log files are designed to be read by humans. Some were made to be parsed by applications. Below are some of examples.
Login failures log
Location: /var/log/faillog
Contains info about login failures. You can view it with the faillog command.
Last logins log
Location: /var/log/lastlog
Contains info about last logins. You can view it with the lastlog command.
Login records log
Location: /var/log/wtmp
Contains login info used by other utilities to find out who’s logged in. To view currently logged in users, use the who command.
This is not an exhaustive list!
You can search the web for more locations relevant to what you’re trying to debug. There is also a longer list here.
3. Viewing logs using GNOME System Log Viewer
The GNOME System Log Viewer provides a simple GUI for viewing and monitoring log files. If you’re running Ubuntu 17.10 or above, it will be called Logs. Otherwise, it will be under the name System Log.
System Log Viewer interface
The log viewer has a simple interface. The sidebar on the left shows a list of open log files, with the contents of the currently selected file displayed on the right.
The log viewer not only displays but also monitors log files for changes. The bold text (as seen in the screenshot above) indicates new lines that have been logged after opening the file. When a log that is not currently selected is updated, it’s name in the file list will turn bold (as shown by auth.log in the screenshot above).
Clicking on the cog at the top right of the window will open a menu allowing you to change some display settings, as well as open and close log files.
There is also a magnifying glass icon to the right of the cog that allows you to search within the currently selected log file.
More information
If you wish to learn more about the GNOME System Log Viewer, you may visit the official documentation.
4. Viewing and monitoring logs from the command line
It is also important to know how to view logs in the command line. This is especially useful when you’re remotely connected to a server and don’t have a GUI.
The following commands will be useful when working with log files from the command line.
Viewing files
The most basic way to view files from the command line is using the cat command. You simply pass in the filename, and it outputs the entire contents of the file: cat file.txt.
This can be inconvenient when dealing with large files (which isn’t uncommon for logs!). We could use an editor, although that may be overkill just to view a file. This is where the less command comes in. We pass it the filename (less file.txt), and it will open the file in a simple interface. From here, we can use the arrow keys (or j/k if you’re familiar with Vim) to move through the file, use / to search, and press q to quit. There are a few more features, all of which are described by pressing h to open the help.
Viewing the start or end of a file
We may also want to quickly view the first or last n number of lines of a file. This is where the head and tail commands come in handy. These commands work much like cat, although you can specify how many lines from the start/end of the file you want to view. To view the first 15 lines of a file, we run head -n 15 file.txt, and to view the last 15, we run tail -n 15 file.txt. Due to the nature of log files being appended to at the bottom, the tail command will generally be more useful.
Monitoring files
To monitor a log file, you may pass the -f flag to tail. It will keep running, printing new additions to the file, until you stop it (Ctrl + C). For example: tail -f file.txt.
Searching files
One way that we looked at to search files is to open the file in less and press /. A faster way to do this is to use the grep command. We specify what we want to search for in double quotes, along with the filename, and grep will print all the lines containing that search term in the file. For example, to search for lines containing “test” in file.txt, you would run grep "test" file.txt.
If the result of a grep search is too long, you may pipe it to less, allowing you to scroll and search through it: grep "test" file.txt | less.
Editing files
The simplest way to edit files from the command line is to use nano. nano is a simple command line editor, which has all the most useful keybindings printed directly on screen. To run it, just give it a filename (nano file.txt). To close or save a file, press Ctrl + X. The editor will ask you if you want to save your changes. Press y for yes or n for no. If you choose yes, it will ask you for the filename to save the file as. If you are editing an existing file, the filename will already be there. Simply leave it as it is and it will save to the proper file.
5. Conclusion
Congratulations, you now have enough knowledge of log file locations, usage of the GNOME System Log Viewer and basic command line commands to properly monitor and trouble-shoot problems that arise on your system.
Further reading
- The Ubuntu Wiki has an article that goes more in-depth into Ubuntu log files.
- This DigitalOcean Community article covers viewing Systemd logs
Was this tutorial useful?
Thank you for your feedback.
The systemd daemon uses a centralized logging system called a journal, which
is managed by the journald daemon. This daemon collects all log entries
generated by the Linux kernel or any other systemd unit service regardless of
their origin and stores them in a format that is easy to access and manipulate.
The journalctl utility is provided for querying and filtering the data held
within the journal as written by the journald service,
While diagnosing or troubleshooting an issue with the server or one of our
services, the first place to look is the log entries in the journal. Due to its
centralized nature, the journal could potentially contain thousands of log
entries that may not be relevant to our current problem. Therefore, it’s
necessary to filter out what we don’t need to quickly find the relevant
information that will help us solve the problem, and that’s what this tutorial
is all about.
🔭 Want to centralize and monitor your Linux logs?
Head over to Logtail and start ingesting your logs in 5 minutes.
Prerequisites
Before you proceed with this tutorial, you will need the following:
- A Linux server that includes a non-root user with
sudoaccess. We tested all
the commands presented in this guide on an Ubuntu 20.04 server. - A basic understanding of how logging works in
Linux.
Step 1 — Permitting a user to view the system logs
By default, a user can only see log entries from systemd services under the
user’s control. If you run the journalctl command without any arguments, you
may see the following message at the top of the output:
Hint: You are currently not seeing messages from other users and the system.
Users in groups 'adm', 'systemd-journal' can see all messages.
Pass -q to turn off this notice.
. . .
This informs you that the output produced by the utility does not include
entries created by system services and those under other users’ control. The way
to ensure that you can view all log messages is by adding the user to an
existing group such as adm or systemd-journal.
Go ahead and add the current user to the systemd-journal group using the
command below:
sudo usermod -a -G systemd-journal <user>
After logging out and logging back in again, you’ll be able to to see all the
available messages, and not just the ones that pertain to the current user.
Step 2 — Querying the journal with Journalctl
In this section, we will query the systemd journal using journalctl, and
view the results in various ways. Enter the command below to see all the logs
collected by the journald daemon:
When used alone without any options, the journalctl command will output all
the journal entries on the system and pipe them through a pager (usually
less). You can also cause journalctl to print its output directly to the
standard output instead of using a pager by including the --no-pager flag.
This is useful if you want to process the data further with text processing
tools like grep, awk, or sed, or redirect the output to a file.
The first line of the output is a header that describes the time range of the
displayed logs:
-- Logs begin at Fri 2022-02-11 15:34:17 UTC, end at Wed 2022-02-16 08:57:01 UTC. --
Below the header, you will see log entries generated by various programs in the
following format and sorted from oldest to newest:
Feb 16 08:59:57 ubuntu-2gb-nbg1-1 vector[74071]: {"appname":"devankoshal","facility":"ftp","hostname":"we.com","message":"Take a breath, let it go, walk away","msgid":"ID911","procid":1211,"severity":"debug","timestamp":"2022-0>
Feb 16 08:59:58 ubuntu-2gb-nbg1-1 vector[74071]: {"appname":"Karimmove","facility":"alert","hostname":"for.de","message":"Maybe we just shouldn't use computers","msgid":"ID528","procid":3542,"severity":"debug","timestamp":"2022>
Feb 16 08:59:59 ubuntu-2gb-nbg1-1 vector[74071]: {"appname":"jesseddy","facility":"authpriv","hostname":"some.net","message":"You're not gonna believe what just happened","msgid":"ID427","procid":7885,"severity":"alert","timest>
Feb 16 09:00:00 ubuntu-2gb-nbg1-1 vector[74071]: {"appname":"benefritz","facility":"daemon","hostname":"up.us","message":"We're gonna need a bigger boat","msgid":"ID220","procid":5116,"severity":"crit","timestamp":"2022-02-16T0>
Feb 16 09:00:01 ubuntu-2gb-nbg1-1 CRON[100444]: pam_unix(cron:session): session opened for user ayo by (uid=0)
Feb 16 09:00:01 ubuntu-2gb-nbg1-1 CRON[100445]: (ayo) CMD (/usr/sbin/logrotate /home/ayo/logrotate.conf --state /home/ayo/custom-state)
Feb 16 09:00:01 ubuntu-2gb-nbg1-1 CRON[100444]: pam_unix(cron:session): session closed for user ayo
Feb 16 09:00:01 ubuntu-2gb-nbg1-1 vector[74071]: {"appname":"benefritz","facility":"uucp","hostname":"make.de","message":"You're not gonna believe what just happened","msgid":"ID783","procid":6478,"severity":"debug","timestamp">
Feb 16 09:00:02 ubuntu-2gb-nbg1-1 vector[74071]: {"appname":"meln1ks","facility":"clockd","hostname":"for.net","message":"#hugops to everyone who has to deal with this","msgid":"ID114","procid":8519,"severity":"debug","timestam>
Feb 16 09:00:03 ubuntu-2gb-nbg1-1 vector[74071]: {"appname":"shaneIxD","facility":"cron","hostname":"make.org","message":"Maybe we just shouldn't use computers","msgid":"ID196","procid":2970,"severity":"err","timestamp":"2022-0>
Each entry starts with a timestamp, the hostname of the machine, the program
that generated the log entry, and its process id. The log message itself comes
afterward.
If you want to print only the last few log entries, you can use the -n option,
which will restrict the printed entries to specified number:
journalctl -n 10 # print the last 10 entries
You can also view incoming log messages in real-time through the -f flag,
which imitates tail -f:
In later sections, we’ll look at more sophisticated ways of filtering the logs
produced by journalctl so that we can find the information we’re looking for
quickly. Let’s discuss customizing the output of the journalctl command first
though.
Step 3 — Customizing the log output format
When parsing the log entries produced by journalctl, it may be useful to
change the format to an easy to parse format like JSON. This is possible by
specifying the json format through the -o option:
journalctl -o json -n 10 --no-pager
This yields the following output:
{"_SELINUX_CONTEXT":"unconfinedn","SYSLOG_FACILITY":"3","_SYSTEMD_SLICE":"system.slice","MESSAGE":"{"appname":"shaneIxD","facility":"mail","hostname":"make.com","message":"A bug was encountered but not in Vector, which doesn't have bugs","msgid":"ID598","procid":2005,"severity":"crit","timestamp":"2022-02-16T23:29:58.696Z","version":1}","_SYSTEMD_UNIT":"vector.service","_SYSTEMD_CGROUP":"/system.slice/vector.service","_HOSTNAME":"ubuntu-2gb-nbg1-1","SYSLOG_IDENTIFIER":"vector","_COMM":"vector","_SYSTEMD_INVOCATION_ID":"5435d275748449c58c051f7c6ecc2e8c","_PID":"74071","_MACHINE_ID":"cee31bed2e414d19ab394c074b55b354","_EXE":"/usr/bin/vector","__REALTIME_TIMESTAMP":"1645054198697037","_STREAM_ID":"a5e2ba59f2d34a829098f6f9db57be9c","_BOOT_ID":"06a58ed252614238b0d09d15161688ed","__MONOTONIC_TIMESTAMP":"481325512974","_TRANSPORT":"stdout","_GID":"997","_UID":"997","PRIORITY":"6","_CAP_EFFECTIVE":"400","_CMDLINE":"/usr/bin/vector","__CURSOR":"s=d96f9da5333a4e1d8394272215ea1917;i=2d710c;b=06a58ed252614238b0d09d15161688ed;m=70113c790e;t=5d82b07264c4d;x=bb9760a144a3a0d1"}
. . .
Notice how detailed this output is compared to the default and how more
information is contained therein. You can also specify json-pretty instead of
json:
journalctl -o json-pretty -n 10 --no-pager
This makes the command output more readable at the cost of screen real-estate:
{
"_PID" : "74071",
"_SYSTEMD_INVOCATION_ID" : "5435d275748449c58c051f7c6ecc2e8c",
"_MACHINE_ID" : "cee31bed2e414d19ab394c074b55b354",
"_SELINUX_CONTEXT" : "unconfinedn",
"PRIORITY" : "6",
"__MONOTONIC_TIMESTAMP" : "481505512370",
"_HOSTNAME" : "ubuntu-2gb-nbg1-1",
"_SYSTEMD_SLICE" : "system.slice",
"SYSLOG_FACILITY" : "3",
"MESSAGE" : "{"appname":"benefritz","facility":"lpr","hostname":"we.com","message":"There's a breach in the warp core, captain","msgid":"ID448","procid":8376,"severity":"crit","timestamp":"2022-02-16T23:32:58.695Z","version":2}",
"SYSLOG_IDENTIFIER" : "vector",
"_SYSTEMD_CGROUP" : "/system.slice/vector.service",
"_UID" : "997",
"_CAP_EFFECTIVE" : "400",
"_STREAM_ID" : "a5e2ba59f2d34a829098f6f9db57be9c",
"_SYSTEMD_UNIT" : "vector.service",
"__CURSOR" : "s=d96f9da5333a4e1d8394272215ea1917;i=2d71f5;b=06a58ed252614238b0d09d15161688ed;m=701bf70bb2;t=5d82b11e0def1;x=529c820dc1c2952c",
"_GID" : "997",
"__REALTIME_TIMESTAMP" : "1645054378696433",
"_COMM" : "vector",
"_EXE" : "/usr/bin/vector",
"_TRANSPORT" : "stdout",
"_CMDLINE" : "/usr/bin/vector",
"_BOOT_ID" : "06a58ed252614238b0d09d15161688ed"
}
Here’s a few of all available formats that control the output produced by
journalctl. You can examine the
complete list here.
cat: includes the log message only.short: the default output format.json: JSON-formatted output containing all available journal fields.json-pretty: prettified JSON output for better readability.verbose: displays the full log entry with all fields.
Now that we have a handle on how to present the journalctl output in different
ways, let’s look at a few ways to narrow down the entries produced by the
utility.
Step 4 — Filtering the journal by boot session
One of the most common ways to filter the journalctl output is by only
including messages that were logged after a specific system boot. For example,
you can view all the logs collected since the most recent boot by executing
journalctl with -b flag:
The program should display an output similar to the one shown below:
-- Logs begin at Fri 2022-02-11 15:34:17 UTC, end at Wed 2022-02-16 09:24:09 UTC. --
Feb 11 15:34:17 ubuntu-2gb-nbg1-1 systemd[1279]: Stopped target Main User Target.
Feb 11 15:34:17 ubuntu-2gb-nbg1-1 systemd[1279]: Stopped target Basic System.
Feb 11 15:34:17 ubuntu-2gb-nbg1-1 systemd[1279]: Stopped target Paths.
Feb 11 15:34:17 ubuntu-2gb-nbg1-1 systemd[1279]: Stopped target Sockets.
Feb 11 15:34:17 ubuntu-2gb-nbg1-1 systemd[1279]: Stopped target Timers.
Feb 11 15:34:17 ubuntu-2gb-nbg1-1 systemd[1279]: dbus.socket: Succeeded.
Feb 11 15:34:17 ubuntu-2gb-nbg1-1 systemd[1279]: Closed D-Bus User Message Bus Socket.
Feb 11 15:34:17 ubuntu-2gb-nbg1-1 systemd[1279]: dirmngr.socket: Succeeded.
Feb 11 15:34:17 ubuntu-2gb-nbg1-1 systemd[1279]: Closed GnuPG network certificate management daemon.
Feb 11 15:34:17 ubuntu-2gb-nbg1-1 systemd[1279]: gpg-agent-browser.socket: Succeeded.
Feb 11 15:34:17 ubuntu-2gb-nbg1-1 systemd[1279]: Closed GnuPG cryptographic agent and passphrase cache (access for web browsers).
Feb 11 15:34:17 ubuntu-2gb-nbg1-1 systemd[1279]: gpg-agent-extra.socket: Succeeded.
Feb 11 15:34:17 ubuntu-2gb-nbg1-1 systemd[1279]: Closed GnuPG cryptographic agent and passphrase cache (restricted).
Feb 11 15:34:17 ubuntu-2gb-nbg1-1 systemd[1279]: gpg-agent-ssh.socket: Succeeded.
Feb 11 15:34:17 ubuntu-2gb-nbg1-1 systemd[1279]: Closed GnuPG cryptographic agent (ssh-agent emulation).
Feb 11 15:34:17 ubuntu-2gb-nbg1-1 systemd[1279]: gpg-agent.socket: Succeeded.
Feb 11 15:34:17 ubuntu-2gb-nbg1-1 systemd[1279]: Closed GnuPG cryptographic agent and passphrase cache.
Feb 11 15:34:17 ubuntu-2gb-nbg1-1 systemd[1279]: pk-debconf-helper.socket: Succeeded.
. . .
This output shows all records in chronological order as before. Before seeing
application logs, you will observe that the oldest records show very low-level
kernel information about the system boot processes.
If you want to print the logs that pertain to a boot session other than the last
one, you can list all boots that journald knows about by executing
journalctl with parameter --list-boots:
You’ll see the program’s output appear on the screen:
-3 9a8ebc63800e4b488b8b0fe90991600c Thu 2021-03-25 18:44:20 UTC—Thu 2021-03-25 18:57:41 UTC
-2 8a0a7c2d722d49f3ba3f411cf2344bb8 Thu 2021-03-25 18:58:11 UTC—Wed 2021-04-14 16:00:31 UTC
-1 0f419686d8744067acd4e7ab962a280b Wed 2021-04-14 16:01:14 UTC—Thu 2021-04-15 14:02:09 UTC
0 dbf7a43ac05f45e39be23091acf434bc Thu 2021-04-15 14:02:32 UTC—Thu 2021-04-15 18:00:31 UTC
The output shows all the available system boots with their offset number, boot
ID (an hexadecimal number), and time range. The value from the offset column can
be used as shown below:
journalctl -b 0 # same as journalctl -b
$ journalctl -b -1 # output logs from the previous boot session
You can also use the boot ID as shown below:
journalctl -b 0f419686d8744067acd4e7ab962a280b
This will show all the log entries that were collected within the boot session
from 2021-04-14 16:01:14 to 2021-04-15 14:02:09.
When dealing with servers that do not often reboot, filtering the logs by system
boot may not be helpful as the current boot session may be the only one
available. Therefore, in the next section, we’ll consider a more granular way to
restrict the log entries outputted by journalctl.
Step 5 — Showing logs within a time range
A typical way to filter system logs is by querying for those that fall before,
after, or between a given period. You can use the --since flag to specify a
lower time limit and the --until for an upper limit. Both flags accepts a
timestamp value that follows the systemd.time
specification.
The following are examples of valid arguments to --since and -until.
2021-11-23 23:02:152021-05-0412:005 hour ago, or32 min agoyesterday,today,now
Let’s go ahead and view today’s records using the command below:
journalctl --since 'today'
You’ll see the program’s output appear on the screen:
-- Logs begin at Fri 2022-02-11 15:34:17 UTC, end at Wed 2022-02-16 21:33:52 UTC. --
Feb 16 00:00:00 ubuntu-2gb-nbg1-1 vector[74071]: {"appname":"ahmadajmi","facility":"local4","hostname":"we.com","message":"#hugops to everyone who has to deal with this","msgid":"ID844","procid":113,"severity":"alert","timestam>
Feb 16 00:00:01 ubuntu-2gb-nbg1-1 systemd[1]: Starting Rotate log files...
Feb 16 00:00:01 ubuntu-2gb-nbg1-1 systemd[1]: Starting Daily man-db regeneration...
Feb 16 00:00:01 ubuntu-2gb-nbg1-1 CRON[79633]: pam_unix(cron:session): session opened for user ayo by (uid=0)
Feb 16 00:00:01 ubuntu-2gb-nbg1-1 CRON[79641]: (ayo) CMD (/usr/sbin/logrotate /home/ayo/logrotate.conf --state /home/ayo/custom-state)
Feb 16 00:00:01 ubuntu-2gb-nbg1-1 CRON[79633]: pam_unix(cron:session): session closed for user ayo
Feb 16 00:00:01 ubuntu-2gb-nbg1-1 systemd[1]: logrotate.service: Succeeded.
Feb 16 00:00:01 ubuntu-2gb-nbg1-1 systemd[1]: Finished Rotate log files.
Feb 16 00:00:01 ubuntu-2gb-nbg1-1 vector[74071]: {"appname":"meln1ks","facility":"ntp","hostname":"make.net","message":"You're not gonna believe what just happened","msgid":"ID477","procid":6062,"severity":"notice","timestamp":>
Feb 16 00:00:01 ubuntu-2gb-nbg1-1 systemd[1]: man-db.service: Succeeded.
Feb 16 00:00:01 ubuntu-2gb-nbg1-1 systemd[1]: Finished Daily man-db regeneration.
Feb 16 00:00:02 ubuntu-2gb-nbg1-1 vector[74071]: {"appname":"shaneIxD","facility":"daemon","hostname":"names.com","message":"Take a breath, let it go, walk away","msgid":"ID74","procid":1031,"severity":"notice","timestamp":"202>
The output may show a lot of records, but you’ll observe that they were all
recorded on the current day. You can also filter logs that fall on a specific
date or between specific dates:
journalctl --since '2022-02-16 21:00:00' --until '2022-02-16 22:00:00'
As you can see, journalctl provides great flexibility when it comes to
filtering records by time. However, the log entries still contain records from
several different services. If you’re only interested in a records from a
specific application, filtering by time only will still yield irrelevant
messages. In the next section, we’ll consider how to restrict the journalctl
output to entries produced by a specific systemd service.
Step 6 — Filtering journal entries by service
If you are interested only in log entries related to a specific systemd unit
service, you can pass the service name to the -u flag. For example, let’s view
the last 10 log messages produced by the sshd service using the command shown
below:
journalctl -u sshd.service -n 10
You’ll see the program’s output appear on the screen:
-- Logs begin at Fri 2022-02-11 15:34:17 UTC, end at Wed 2022-02-16 22:03:52 UTC. --
Feb 16 21:56:48 ubuntu-2gb-nbg1-1 sshd[61254]: Received disconnect from 122.194.229.45 port 19218:11: [preauth]
Feb 16 21:56:48 ubuntu-2gb-nbg1-1 sshd[61254]: Disconnected from 122.194.229.45 port 19218 [preauth]
Feb 16 21:57:39 ubuntu-2gb-nbg1-1 sshd[61258]: Invalid user test3 from 45.112.242.67 port 55536
Feb 16 21:57:39 ubuntu-2gb-nbg1-1 sshd[61258]: Received disconnect from 45.112.242.67 port 55536:11: Bye Bye [preauth]
Feb 16 21:57:39 ubuntu-2gb-nbg1-1 sshd[61258]: Disconnected from invalid user test3 45.112.242.67 port 55536 [preauth]
Feb 16 21:57:52 ubuntu-2gb-nbg1-1 sshd[61260]: Invalid user wedding from 51.140.185.84 port 51118
Feb 16 21:57:52 ubuntu-2gb-nbg1-1 sshd[61260]: Received disconnect from 51.140.185.84 port 51118:11: Bye Bye [preauth]
Feb 16 21:57:52 ubuntu-2gb-nbg1-1 sshd[61260]: Disconnected from invalid user wedding 51.140.185.84 port 51118 [preauth]
Feb 16 21:59:41 ubuntu-2gb-nbg1-1 sshd[61265]: Accepted publickey for ayo from 217.138.222.109 port 59800 ssh2: RSA SHA256:8Gdo7Q35uybWwTaGAoWrQXowqn0MEGaErerTlpR7nTM
Feb 16 21:59:41 ubuntu-2gb-nbg1-1 sshd[61265]: pam_unix(sshd:session): session opened for user ayo(uid=1000) by (uid=0)
From the output above, you can observe that only the records that pertain to
sshd are displayed. If you need to view the logs for more than one service,
you can repeat the -u flag in the command with the names of different
services.
journalctl -u rsyslog.service -u nginx.service --since today
The entries produced from both services will be merged and presented in
chronological order. You can also use the --since and --until flags to
narrow down the results further as you see fit.
Step 7 — Filtering journal entries by priority level
Each journal record has a well-defined structure that also includes message
priority. The journalctl command allows filtering records by message priority,
which are the same as the standard syslog priority levels (listed in
descending order):
{
emerg: 0,
alert: 1,
crit: 2,
error: 3,
warning: 4,
notice: 5,
info: 6,
debug: 7
}
Here’s an explanation of the levels above:
emerg: the system is unusable.alert: action must be taken immediately.crit: critical conditions.err: error conditions.warning: warning conditions.notice: normal, but significant condition.info: informational message.debug: messages that are useful for debugging.
You can specify the priority name or its corresponding number value when
filtering log records with journalctl by using the -p option:
journalctl -p err
# or
$ journalctl -p 3
The command above will display all messages prioritized at err level or above.
This means that the output will not contain messages logged at warning level
or below.
-- Logs begin at Fri 2022-02-11 15:34:17 UTC, end at Wed 2022-02-16 22:58:21 UTC. --
Feb 11 15:41:23 ubuntu-2gb-nbg1-1 sudo[909]: pam_unix(sudo:auth): auth could not identify password for [ayo]
Feb 11 15:41:23 ubuntu-2gb-nbg1-1 sudo[909]: ayo : user NOT in sudoers ; TTY=pts/0 ; PWD=/home/ayo ; USER=root ; COMMAND=/usr/bin/dnf install fish
Feb 11 15:41:34 ubuntu-2gb-nbg1-1 sudo[911]: ayo : user NOT in sudoers ; TTY=pts/0 ; PWD=/home/ayo ; USER=root ; COMMAND=/usr/bin/dnf update
Feb 11 15:44:31 ubuntu-2gb-nbg1-1 sshd[976]: fatal: Timeout before authentication for 1.15.86.71 port 41058
Feb 11 20:38:11 ubuntu-2gb-nbg1-1 sshd[3300]: error: kex_exchange_identification: Connection closed by remote host
Feb 11 20:58:23 ubuntu-2gb-nbg1-1 sshd[3348]: error: kex_exchange_identification: Connection closed by remote host
Feb 12 00:44:08 ubuntu-2gb-nbg1-1 sshd[4536]: error: kex_exchange_identification: Connection closed by remote host
Feb 12 01:22:43 ubuntu-2gb-nbg1-1 sshd[4622]: error: kex_exchange_identification: Connection closed by remote host
. . .
This feature makes it easy to remove irrelevant information when troubleshooting
a problem so that the higher priority messages can take center stage.
Step 8 — Cleaning up old journal entries
The journalctl command also provides control over how much space is used up by
the journal, and when to clean up older entries. You can use the --disk-usage
flag to discover how much is currently being used up:
You should see the following output:
Archived and active journals take up 1.8G in the file system.
If you want to shrink your journal to a certain size, you can use the
--vacuum-size option as shown below:
sudo journalctl --vacuum-size=500M # shrink journal to 500 MB.
You’ll see the program’s output appear on the screen:
Deleted archived journal /var/log/journal/cee31bed2e414d19ab394c074b55b354/system@d96f9da5333a4e1d8394272215ea1917-00000000001705f4-0005d7c8815c6001.journal (128.0M).
Deleted archived journal /var/log/journal/cee31bed2e414d19ab394c074b55b354/system@d96f9da5333a4e1d8394272215ea1917-000000000018a118-0005d7ccffbf428b.journal (128.0M).
Deleted archived journal /var/log/journal/cee31bed2e414d19ab394c074b55b354/system@d96f9da5333a4e1d8394272215ea1917-00000000001a39bc-0005d7d1621e9d0e.journal (128.0M).
Deleted archived journal /var/log/journal/cee31bed2e414d19ab394c074b55b354/system@d96f9da5333a4e1d8394272215ea1917-00000000001bd232-0005d7d5977c6dd9.journal (128.0M).
Deleted archived journal /var/log/journal/cee31bed2e414d19ab394c074b55b354/system@d96f9da5333a4e1d8394272215ea1917-00000000001d6237-0005d7d9614516c8.journal (128.0M).
Deleted archived journal /var/log/journal/cee31bed2e414d19ab394c074b55b354/system@d96f9da5333a4e1d8394272215ea1917-00000000001ef1f4-0005d7dd2e4431c9.journal (128.0M).
Deleted archived journal /var/log/journal/cee31bed2e414d19ab394c074b55b354/system@d96f9da5333a4e1d8394272215ea1917-00000000002081d7-0005d7e0f8261d1a.journal (128.0M).
Deleted archived journal /var/log/journal/cee31bed2e414d19ab394c074b55b354/system@d96f9da5333a4e1d8394272215ea1917-000000000022130a-0005d7e4c5032aa2.journal (128.0M).
Deleted archived journal /var/log/journal/cee31bed2e414d19ab394c074b55b354/system@d96f9da5333a4e1d8394272215ea1917-000000000023ac43-0005d7e9129c10c1.journal (128.0M).
Deleted archived journal /var/log/journal/cee31bed2e414d19ab394c074b55b354/system@d96f9da5333a4e1d8394272215ea1917-000000000025457d-0005d7ecc6945d7e.journal (128.0M).
Deleted archived journal /var/log/journal/cee31bed2e414d19ab394c074b55b354/system@d96f9da5333a4e1d8394272215ea1917-000000000026d5ce-0005d7f0c66f8cfe.journal (128.0M).
Vacuuming done, freed 1.3G of archived journals from /var/log/journal/cee31bed2e414d19ab394c074b55b354.
Vacuuming done, freed 0B of archived journals from /run/log/journal.
Vacuuming done, freed 0B of archived journals from /var/log/journal.
As you can see, the journal was shrunk to 500 MB after log entries totalling 1.3
GB in size were deleted from the archive. An alternative to --vacuum-size is
the --vacuum-time option which allows you to delete logs older than a certain
period of time.
For example, you can delete the entries that were created more than one month
ago through the command below:
sudo journalctl --vacuum-time=1month
You can also limit the storage space that the journal takes up by editing the
following options
in the /etc/systemd/journald.conf file:
SystemMaxUse=andRuntimeMaxUse=: the maximum amount of space that the
journal should take up in a persistent storage (/var/log/journal) and
in-memory storage (/run/log/journal) respectively.SystemKeepFree=andRuntimeKeepFree=: defines the percentage of disk space
that should be kept free for other uses.SystemMaxFileSize=andRuntimeMaxFileSize=: controls how large journal
entries should grow before being rotated.SystemMaxFiles=andRuntimeMaxFiles=: controls the maximum number of
journal files to keep.
Conclusion
In this article, you learned about the systemd journal and how to manage it
through the journalctl command. We started by discussing what the journal is
for, and then described how to view and filter its entries in various ways. To
learn more about the systemd journal and the journalctl command, consult the
official documentation
or type man journalctl in your terminal.
Thanks for reading, and happy logging!
Centralize all your logs into one place.
Analyze, correlate and filter logs with SQL.
Create actionable
dashboards.
Share and comment with built-in collaboration.
Got an article suggestion?
Let us know
Next article
How to Set Up Centralized Logging on Linux with Rsyslog
Learn how to set up a centralized logging on linux with rsyslog
→
Explore more
Learn everything you want to know about logging. Go from basics to best practices in no time.
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Если вы столкнулись с проблемами в работе сервера, первое, что нужно сделать — посмотреть логи Linux. В системный журнал записываются диагностические сообщения, поступающие от различных компонентов операционной системы, таких как ядро или службы, поэтому с большой долей вероятности причина сбоев будет найдена.
Каждое сообщение генерируется в результате возникновения какого-либо события в операционной системе. Событием может быть остановка службы, авторизации пользователя в системе или неполадки в работе приложения. События имеют определенный приоритет, в зависимости от степени критичности. В Linux различают следующие типы событий:
- emerg — авария, наивысший приоритет;
- alert — тревога;
- crit — критическое событие;
- err — ошибка;
- warn — внимание;
- notice — уведомление;
- info — информационное сообщение;
- debug — отладочная информация;
На сегодняшний день в Linux основными службами сбора логов являются rsyslog и systemd-journald, они работают независимо друг от друга и входят в состав большинства современных дистрибутивов.
rsyslog
Журналы службы находятся в директории “/var/log/” в виде обычных текстовых файлов. В зависимости от типа события, сообщения записываются в разные файлы. Например файл “/var/log/auth.log” содержит информацию о входе пользователей в систему, а в файл “/var/log/kern.log” записываются сообщения ядра. В разных дистрибутивах названия файлов могут отличаться, поэтому для точного понимания куда именно происходит запись сообщений рассмотрим файл конфигурации “/etc/rsyslog.d/50-default.conf”.
Правила описывают место хранения логов в зависимости от типа сообщения. В левой части строки указан тип сообщения в формате “[Источник].[Приоритет]”, а в правой части имя файла журнала. При записи типа сообщения можно применять символ “*”, обозначающий любое значение или параметр “none”, обозначающий исключение из списка. Рассмотрим более подробно первые два правила.
“auth,authpriv.* /var/log/auth.log”
“*.*;auth,authpriv.none -/var/log/syslog”
Первое правило означает, что все сообщения принятые от механизма авторизации будут записаны в файл “/var/log/auth.log”. В этом файле будут зарегистрированы все попытки входа пользователей в систему, как удачные так и не удачные. Второе правило говорит о том, что все сообщения, кроме тех, которые связаны с авторизацией будут записаны в файл “/var/log/syslog”. Именно к этим файлам приходится обращаться наиболее часто. Следующие правила определяют место хранения журналов ядра “kern.*” и почтовой службы “mail.*”
Журналы логов можно открыть любой утилитой для просмотра текста, например less, cat, tail. Откроем файл “/var/log/auth.log”
less /var/log/auth.log
Каждая строка файла является отдельным сообщением, поступившим от приложения или службы. Все сообщения, независимо от источника имеют единый формат и состоят из пяти частей. Рассмотрим их на примере выделенного сообщения на скриншоте.
- Дата и время регистрации сообщения — “Feb 12 06:18:33”
- Имя компьютера, с которого пришло сообщение — “vds”
- Имя программы или службы, к которой относится сообщение — “sshd”
- Идентификатор процесса, отправившего сообщение — [653]
- Текст сообщения — “Accepted password for mihail from 188.19.42.165 port 2849 ssh2”
Это был пример успешного подключения по ssh.
А так выглядит неудачная попытка:
В этом файле также фиксируется выполнение команд с повышенными правами.
Откроем файл /var/log/syslog
На скриншоте выделено сообщение о выключении сетевого интерфейса.
Для поиска нужной информации в больших текстовых файлах можно использовать утилиту grep. Найдем все сообщения от службы pptpd в файле “/var/log/syslog”
grep 'pptpd' /var/log/syslog
Во время диагностики можно использовать утилиту tail, которая выводит последние строки в файле. Команда “tail -f /var/log/syslog” позволит наблюдать запись логов в реальном времени.
Служба rsyslog является очень гибкой, высокопроизводительной и может использоваться для сбора логов как на локальных системах, так и на уровне предприятия. Полную документацию можно найти на официальном сайте https://www.rsyslog.com/
Запись логов происходит непрерывно и размер файлов постоянно растет. Механизм ротации обеспечивает автоматическое архивирование старых журналов и создание новых. В зависимости от правил, обработка журналов может выполняться ежедневно, еженедельно, ежемесячно или при достижении файлом определенного размера. По мере создания новых архивов, старые могут быть просто удалены или предварительно отправлены по электронной почте. Ротация выполняется утилитой logrotate. Основная конфигурация находится в файле “/etc/logrotate.conf”, также обрабатывается содержимое файлов в директории “/etc/logrotate.d/”
Новые правила можно записывать в основной файл конфигурации, но более правильным будет создание отдельного файла в директории “/etc/logrotate.d/” По умолчанию в директории уже содержится несколько файлов.
Рассмотрим файл “/etc/logrotate.d/rsyslog”, который содержит правила ротации для журналов службы rsyslog.
В начале правила указывается путь к файлу журнала, затем в фигурных скобках перечисляются директивы.
- rotate 7 — необходимо постоянно хранить 7 файлов
- daily — ежедневно будет создаваться новый файл
- compress — старые файлы необходимо архивировать.
На скриншоте видно, что в каталоге “/var/log/” находится основной журнал “syslog” и семь архивов, что соответствует правилам ротации.
Более подробное описание по настройке утилиты logrotate можно найти в мануале, выполнив команду “man logrotate”
journald
Служба сбора логов systemd-journald является частью системы инициализации systemd. Файлы журнал хранятся в директории “/var/log/journal/” в специальном формате и могут быть открыты с помощью утилиты journalctl. Формат записей такой же как у службы rsyslog.
Команда journalctl без параметров выводит на экран все записи, но учитывая, что объем журнала может достигать нескольких гигабайт, такой способ просмотра не подходит для практического применения. Рассмотрим некоторые опции утилиты.
- вывод записей с момента последней загрузки
journalctl -b - вывод записей за определенный период времени
journalctl -S "2020-02-17 12:00" -U "2020-02-17 12:10" - вывод записей, принятых от определенной службы
journalctl -u pptpd - вывод сообщений ядра
journalctl -k - вывод сообщений с определенным приоритетом, в данном случае будут выведены ошибки и более высокие приоритеты(crit, alert, emerg).
journalctl -p err - вывод сообщений в реальном времени
journalctl -f
Для более гибкого поиска опции можно совмещать. Выведем все ошибки службы pptpd
journalctl -u pptpd -p err
Если в качестве аргумента указать путь к исполняемому файлу, утилита выведет все сообщения, отправленные этим файлом. Выведем сообщения, отправленные файлом “/usr/bin/sudo” начиная с 04:15 18-го февраля 2020 года. Фактически будут выведены все команды, выполненные с повышенными правами.
journalctl -S "2020-02-18 04:15" /usr/bin/sudo
Для того, чтобы узнать сколько места на диске занимают файлы журнала, выполним команду
journalctl --disk-usage
Для ограничения объема журнала размером 1Gb выполним команду
journalctl --vacuum-size=1G
Открытие бинарных файлов
В заключении рассмотрим несколько специальных файлов в директории “/var/log/”, в которых регистрируются попытки входа пользователей в систему. Это бинарные файлы, которые могут быть открыты только специальными утилитами.
/var/log/wtmp — содержит информацию об успешном входе пользователей в систему, для открытия используется утилита last
/var/log/btmp — в файле регистрируются все неудачные попытки входа в систему, открывается командой lastb с повышенными правами. Параметр -n определяет количество выводимых строк начиная с конца файла.
/var/log/lastlog — содержит время последнего входа для каждой учетной записи, может быть открыт одноименной утилитой lastlog
