Docker как посмотреть ошибки

Knowledge on logging with Docker can be very helpful for day-to-day system administration activities, be it troubleshooting or basic monitoring. Without tracking logs on Docker, mitigating issues would be a lot more difficult when investigating anomalies.

This write-up explores some important information on that area to make it easier to understand how to manage Docker associated log files on your Linux system.

Let me start by exploring the most basic parts and gradually go deeper into some specifics.

You can use the docker logs command to fetch whatever is going on with the service inside a running container.

The syntax is simple:

docker logs container_name_or_ID

You probably already know that you can use the docker ps command to view the running containers’ name and ID.

Let’s take a real-world example. I have deployed Nextcloud with Docker. The container is named nextcloud, unsurprisingly.  

Now, to view the logs of the container named nextcloud, use:

docker logs nextcloud

Here’s a truncated version of the output of the above command:

172.18.0.2 - - [23/Jul/2021:19:36:09 +0000] "HEAD /.env HTTP/1.1" 302 1571 "-" "python-requests/2.26.0"
172.18.0.2 - - [23/Jul/2021:19:49:52 +0000] "HEAD /c99.php HTTP/1.1" 302 1565 "-" "python-requests/2.26.0"
172.18.0.2 - - [24/Jul/2021:16:25:23 +0000] "HEAD /backup.tar HTTP/1.1" 302 1571 "-" "python-requests/2.26.0"
172.18.0.2 - - [24/Jul/2021:16:25:24 +0000] "HEAD /backup.zip HTTP/1.1" 302 1569 "-" "python-requests/2.26.0"
172.18.0.2 - - [25/Jul/2021:20:36:01 +0000] "GET / HTTP/1.1" 302 1590 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; BW/1.1; bit.ly/2W6Px8S) Chrome/84.0.4147.105 Safari/537.36"
172.18.0.2 - - [25/Jul/2021:20:36:07 +0000] "GET /custom_apps/richdocumentscode/proxy.php?req=/hosting/capabilities HTTP/1.1" 200 721 "-" "Nextcloud Server Crawler"
172.18.0.2 - - [25/Jul/2021:20:36:04 +0000] "GET /login HTTP/1.1" 200 7501 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; BW/1.1; bit.ly/2W6Px8S) Chrome/84.0.4147.105 Safari/537.36"

Too much of logs? The good thing is that there are various options that make viewing docker logs a lot easier, just like logs in Linux.

Tail Docker logs to view only certain number of lines

Docker logs command has --tail attribute that can be used in a fashion similar to the tail command.

Which means, you can use it to display only a certain number of lines of Docker logs from the end.

For example, to view the last 50 lines of a container, you can use:

docker logs --tail 50 container_name_or_ID

You may use the above command in this fashion as well

docker logs -n=50 container_name_or_ID

Viewing Docker logs in real-time on a live container

You can totally view the container logs in real time. To “follow” the logs, use the --follow or the -f attribute.

docker logs -f container_name_or_ID

When you run docker logs with follow option, you’ll notice that the new log lines will be reported from the container as time progresses.

To exit, you can simply use the Ctrl+C key combination and drop back to the terminal prompt.

Most of the options in the docker logs sub-command can be combined to produce a desired result.

For example, docker logs -f will flood the screen with all the logs from the beginning. You wouldn’t want that. What you can do is to combine both tail and follow options like this:

docker logs -f --tail 20 container_name_or_ID

The above command will show the last 20 lines of the logs and then follow the logs in real time.

When updating your app containers with scaling, the above command can be very helpful before you remove an older container.

View timestamp in Docker logs

If you also want to see the timestamp of the log entries, you can use the -t option.

docker logs -t container_name_or_ID

This can easily be combined with other options like --follow and --tail.

The command below will show the last 10 lines of the log file of a given container with timestamps attached to each of them

docker logs -n=10 -t container_name_or_ID

Viewing Docker logs in a specified time period

When you are investigating an issue, time can be a critical factor, in which case the --since and --until flags could prove very helpful.

For example, if you are investigating an issue that occurred in the past 24 hours, the following command will show the logged contents of whatever happened only during that time:

docker logs --since 1440m -t container_name_or_ID

1440m in the above command denotes 24×60 minutes. Adding the -t or the --timestamps flag ensures the reported lines get labeled with a timestamp for you to comprehend incidents or errors even more easily. I’d suggest adding it when inquiring any container.

Similarly, if you require the log details after the first 24 hours of initiating deployment, the equivalent command would be:

docker logs --until 1440m -t nextcloud

Other than specifying time in minutes, the above two flags can also be used with a specific timestamp that -t generates. The format is like 2021-07-28T06:20:00.

docker logs --since 2021-07-28 -t container_name_or_ID

The above command will show the logs from 28th July 2021.

For a complete overview of docker logs, you can also check its man page with man docker-logs.

Accessing Docker logs from within the container

In some interesting use-cases, you may want to go for a hybrid approach, where you access the application specific logs from within the containers.

To do that, you enter the Docker container:

docker exec -it container_name_or_ID sh

And then you can use the regular Linux tools to get application related logs. One approach would be to get the process ID of the desired service using the ps command:

ps aux

Get the details on this process:

cat /proc/PID_of_process/fd/1

Docker system service logs

If you wish to see the logs of the Docker service itself on the host, you can use journalctl if you are on Ubuntu 16.04 or later:

sudo journalctl -u docker

Where are Docker logs stored?

You should find the docker logs in the /var/lib/docker/containers directory on the host system. This directory contains the log files related to all containers in their individual directories. You can identify the directories with the container ID.

I used the docker ps to find that 70f19fde9076 is the container ID of the Nextcloud container.

If I look into the directory where docker logs are stored:

[email protected]:~$ sudo ls -lh /var/lib/docker/containers

total 16K
drwx------ 4 root root 4.0K Jul 27 18:48 0efe12b28562c42619e533ad5f524d56740a7a3739d9e082c758bac95ae2a46f
drwx------ 4 root root 4.0K Jul 27 18:57 12c55f365f93ecb7f91e40bc130ddc2409216a61bbb244cd045a22b4513416d3
drwx------ 4 root root 4.0K Jul 27 18:58 70f19fde907672b9a6e5ff3b7db0c9ecbcb68d419712cb04d03d77694cd2ca4e
drwx------ 4 root root 4.0K Jul 27 18:57 a436399ef16a3efc0a909b9c3f725938e595e0b8fd34644b13f28b2c9bcb4ed7

There are currently four containers running, and you can see that the third one matches the one we need to look into (it begins with 70f19fde9076).

If you check the contents of this directory, you can see that our log file sits right there!

[email protected]:~$ sudo ls -lh /var/lib/docker/containers/70f19fde907672b9a6e5ff3b7db0c9ecbcb68d419712cb04d03d77694cd2ca4e

total 192K
-rw-r----- 1 root root 150K Jul 27 18:58 70f19fde907672b9a6e5ff3b7db0c9ecbcb68d419712cb04d03d77694cd2ca4e-json.log
drwx------ 2 root root 4.0K Jul 19 17:10 checkpoints
-rw------- 1 root root 5.5K Jul 27 18:58 config.v2.json
-rw-r--r-- 1 root root 1.5K Jul 27 18:58 hostconfig.json
-rw-r--r-- 1 root root   13 Jul 27 18:58 hostname
-rw-r--r-- 1 root root  198 Jul 27 18:58 hosts
drwx------ 3 root root 4.0K Jul 19 17:10 mounts
-rw-r--r-- 1 root root   79 Jul 27 18:58 resolv.conf
-rw-r--r-- 1 root root   71 Jul 27 18:58 resolv.conf.hash

Therefore, the corresponding log file is 70f19fde907672b9a6e5ff3b7db0c9ecbcb68d419712cb04d03d77694cd2ca4e-json.log which is the same file that you were reading when I ran the docker logs command in the beginning.

Let me quickly verify that:

[email protected]:~$ sudo cat /var/lib/docker/containers/70f19fde907672b9a6e5ff3b7db0c9ecbcb68d419712cb04d03d77694cd2ca4e/70f19fde907672b9a6e5ff3b7db0c9ecbcb68d419712cb04d03d77694cd2ca4e-json.log

{"log":"172.18.0.2 - - [23/Jul/2021:19:36:09 +0000] "HEAD /.env HTTP/1.1" 302 1571 "-" "python-requests/2.26.0"n","stream":"stdout","time":"2021-07-23T19:36:09.837857567Z"}
{"log":"172.18.0.2 - - [23/Jul/2021:19:49:52 +0000] "HEAD /c99.php HTTP/1.1" 302 1565 "-" "python-requests/2.26.0"n","stream":"stdout","time":"2021-07-23T19:49:52.996108799Z"}
{"log":"172.18.0.2 - - [24/Jul/2021:16:25:23 +0000] "HEAD /backup.tar HTTP/1.1" 302 1571 "-" "python-requests/2.26.0"n","stream":"stdout","time":"2021-07-24T16:25:23.445225166Z"}
{"log":"172.18.0.2 - - [24/Jul/2021:16:25:24 +0000] "HEAD /backup.zip HTTP/1.1" 302 1569 "-" "python-requests/2.26.0"n","stream":"stdout","time":"2021-07-24T16:25:24.772881041Z"}
{"log":"172.18.0.2 - - [25/Jul/2021:20:36:01 +0000] "GET / HTTP/1.1" 302 1590 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; BW/1.1; bit.ly/2W6Px8S) Chrome/84.0.4147.105 Safari/537.36"n","stream":"stdout","time":"2021-07-25T20:36:03.678967877Z"}
{"log":"172.18.0.2 - - [25/Jul/2021:20:36:07 +0000] "GET /custom_apps/richdocumentscode/proxy.php?req=/hosting/capabilities HTTP/1.1" 200 721 "-" "Nextcloud Server Crawler"n","stream":"stdout","time":"2021-07-25T20:36:07.404618408Z"}
{"log":"172.18.0.2 - - [25/Jul/2021:20:36:04 +0000] "GET /login HTTP/1.1" 200 7501 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; BW/1.1; bit.ly/2W6Px8S) Chrome/84.0.4147.105 Safari/537.36"n","stream":"stdout","time":"2021-07-25T20:36:07.619020402Z"}

The log contents are the same, but since I’m reading the log file directly from the host, this has a lot of jargon compared to the docker logs command used earlier.

Enabling Log Rotation for Docker (JSON)

By default, for JSON file based logging, log rotation on Docker is disabled.

This may pose a problem with the disk space if the log files grows bigger in size. Our Docker based Ghost instance saw its log files going up to 20 GB in size. You would want to avoid such situation by enabling log rotation.

To enable log rotation for Docker, edit the /etc/docker/daemon.json file:

sudo nano /etc/docker/daemon.json

Append the following lines and save the file:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3" 
  }
}

Restart Docker daemon:

sudo systemctl restart docker

The above setup is of course relevant to a generic Docker installation, which has JSON logging set as default. So do note that every other logging mechanism (listed below) has its own way of implementing rotation.

A quick word about logging drivers

Did you notice “json” in the log filename from earlier? That’s because JSON File is the default logging driver on Docker.

Apart from JSON, there are many other logging mechanisms to choose from as well:

• Local file
• Logentries
• Graylog Extended Format(GELF)
• Syslog
• Amazon Cloudwatch Logs(AWS)
• Event Tracing in Windows(ETW)
• Fluentd
• Google Cloud Project Logs(GCP)
• Journald
• Splunk
• Logstash

For large-scale deployments, GELF, AWS, GCP and Fluentd are recommended as they implement a centralized approach. But for small scale, JSON does the job just fine, while Local, Syslog and Journald are also quite suitable. Syslog and Logstash are particularly very helpful if complex log analysis becomes a requirement. ETW is Windows specific, whereas Splunk is focused on remote logging.

Starting from Docker Engine 20.10, a new feature called “dual-logging” has been introduced that ensures executing the docker logs command as well as performing log rotation irrespective of the logging driver in effect.

Summary

In this article, I’ve begun with the basics of how to view the logs for any Docker container, both at the container and host-system level. I’ve also showed you how to locate the physical location of container-specific log files residing on your host-system. I briefly showed you how to check the logs for the Docker service on the host itself.

Later, I’ve discussed logging drivers with focus on the default mechanism and also highlighted the new dual logging feature.

I hope these pointers help you in your daily monitoring or learning activities. Any more thoughts and suggestions are more than welcome! Please share them in the comment section below.

Docker automatically aggregates container standard output and error streams (stdout/stderr) into log feeds which are retained by the Docker daemon. You can easily monitor logs to understand what’s going on in your containers.

The logs contain the output you’d see in your terminal when attached to a container in interactive mode (-it). Logs will only be available if the foreground process in your container actually emits some output. You should make sure your containers log errors to stderr so Docker commands are able to expose them.

You can view container logs at any time during a container’s lifecycle. When a container’s running, you can stream logs in realtime. For stopped containers, you can access all the logs captured prior to the termination.

Viewing Container Logs

To view container logs, use the docker logs command:

docker logs my-container

Replace my-container with the name or ID of the container you want to inspect. You can use docker ps -a to get the IDs and names of your containers.

The logs command prints the container’s entire log output to your terminal. The output will not be continuous. If you’d like to keep streaming new logs, add the --follow flag to the command. This is equivalent to using tail -f with regular log files on your machine.

Customising What’s Displayed

The docker logs command supports several flags that let you adjust its output:

• –timestamps – Display complete timestamps at the start of each log line.
• –until and --since – These flags let you fetch lines logged during a particular time period. Either pass a complete timestamp (2021-04-30T20:00:00Z) or a friendly relative time (e.g. 1h = 1 hour ago).
• –tail – Fetch a given number of lines from the log. --tail 10 will display the last ten lines logged by the container.
• –details – This is a special flag which adds extra information to the log output, based on the options passed to the logging driver. We’ll look at logging drivers in the next section. Typical values displayed with --details include container labels and environment variables.

You can combine these flags to get logs in the format you require. The until, since and tail flags won’t take effect if you’re using follow to continuously stream log data.

Docker Logging Drivers

Docker collects and stores container logs using one of several logging drivers. You can set the active logging driver on a per-container basis. When no logging driver is specified, Docker uses the json-file driver.

This driver stores container logs in a JSON file. This format is fairly human-readable and can be readily consumed by third-party tools. If you’re not going to access log files directly, switching to the local driver will save you some storage space. It uses a custom log storage format.

Other built-in log drivers include syslog (write to the syslog daemon running on your machine), journald (use a running journald instance) and fluentd (to use a fluentd daemon). Drivers are also available for Amazon CloudWatch, Google Cloud Platform, Event Tracing for Windows and other log monitoring solutions.

Docker supports third-party logging drivers via plugins. You can find drivers on Docker Hub. To install a plugin driver, run docker plugin install plugin-name. You’ll then be able to reference it as a logging driver as plugin-name.

Specifying a Logging Driver

You can specify the logging driver for a container by passing the --log-driver flag to docker run:

docker run --log-driver systemd my-image:latest

You can change the default logging driver globally by updating your Docker daemon configuration. Edit (or create) /etc/docker/daemon.json. Set the log-driver key to the name of a logging driver. Docker will use this driver for all containers created without a --log-driver flag.

{
    "log-driver": "systemd"
}

Many logging drivers come with their own configuration options. These are set using the --log-opts container flag, or log-opts in daemon.json. Here’s an example relevant to the default json-file driver. It instructs Docker to rotate log files once they’re larger than 8MB. Only five files will be retained at any time.

docker run

docker run --log-driver json-file --log-opts max-size=8M --log-opts max-file=5

/etc/docker/daemon.json

{
    "log-driver": "json-file",
    "log-opts": {
        "max-size": "8M",
        "max-file": 5
    }
}

Driver Delivery Modes

Logs can be delivered in either blocking or non-blocking modes. Docker defaults to blocking delivery. Logs from the container will be sent to the driver immediately. This guarantees log delivery but could impact performance. The application will wait until the log write is complete. This can cause a perceptible delay if the logging driver is busy.

When in non-blocking mode, Docker writes logs to an in-memory buffer. The container doesn’t need to wait for the logging driver to complete its write. This can significantly improve performance on active machines with slow storage.

The tradeoff with non-blocking mode is the possibility of lost logs. This can occur when logs are emitted more quickly than the driver can process them. The in-memory buffer could be filled, causing cached logs to be cleared before they’ve been handed to the driver.

You can enable non-blocking delivery by setting the mode logging option, either with --log-opts or daemon.json. You can set the size of the in-memory log buffer with the max-buffer-size option. Setting this high reduces the risk of lost logs, provided you’ve got sufficient RAM available.

docker run --log-opt mode=non-blocking --log-opt max-buffer-size=8M my-image:latest

Logging Best Practices

Your containers should work with Docker’s logging system wherever possible. Emitting logs to stdout and stderr allows Docker and other tools to aggregate them in a standardised way.

Log output doesn’t need to include timestamps. Docker’s logging drivers will automatically record the time at which an event occurred.

Sometimes you might have complex logging requirements that docker logs alone can’t satisfy. If that’s the case, you might need to implement your own logging solution within your container. You can store logs directly on the filesystem, using a Docker volume, or call an external API service.

Some stacks call for a dedicated logging container that sits alongside your application containers. The logging container, often called a “sidecar”, reads temporary log files which your application containers create in a shared Docker volume. The sidecar handles the aggregation of these logs into a format which can be uploaded to a log monitoring service.

This approach can be useful for more complex deployments, although it’s trickier to setup and scale. It typically leaves you without the immediate convenience of Docker’s built-in log commands.

Summary

Docker has versatile log monitoring capabilities provided by a suite of logging drivers. Each container can use a unique logging driver, letting you store logs in a format appropriate to each app’s requirements.

Logs include anything emitted by a container’s standard output streams. You can use echo, print, console.log() or your programming language’s equivalent to add lines to the docker logs output. Logs are retained until your container is deleted with docker rm.