1. Overview
Docker is an operating system-level virtualization platform that allows us to host applications in containers. Furthermore, it facilitates the separation of applications and infrastructure for fast software delivery.
Log files generated by Docker containers contain a variety of useful information. Whenever an event occurs, the Docker container creates log files.
Docker generates the logs to the STDOUT or STDERR, including log origin, output stream data, and timestamp. Debugging and finding the root cause of problems can be done using log files.
In this tutorial, we’ll look into accessing the Docker logs in different ways.
2. Understanding Docker Logs
In Docker, primarily, there are two types of log files. The Docker daemon logs provide insight into the Docker service’s overall status. The Docker container logs cover all the logs related to a particular container.
We’ll primarily explore the different commands to access the Docker container logs. We’ll examine the container logs using the docker logs command and by directly accessing the logs file on the system.
The log files are useful to debug problems because they provide details about what occurred. By analyzing Docker logs, we can diagnose and troubleshoot issues faster.
3. Using the docker logs Command
Before we move forward, lets’ first run a sample Postgres Docker container:
$ docker run -itd -e POSTGRES_USER=baeldung -e POSTGRES_PASSWORD=baeldung -p 5432:5432 -v /data:/var/lib/postgresql/data --name postgresql-baeldung postgres
Unable to find image 'postgres:latest' locally
latest: Pulling from library/postgres
214ca5fb9032: Pull complete
...
95df4ec75c64: Pull complete
Digest: sha256:2c954f8c5d03da58f8b82645b783b56c1135df17e650b186b296fa1bb71f9cfd
Status: Downloaded newer image for postgres:latest
bce34bb3c6175fe92c50d6e5c8d2045062c2b502b9593a258ceb6cafc9a2356a
To illustrate, let’s check out the containerId of the postgresql-baeldung container:
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
bce34bb3c617 postgres "docker-entrypoint.s…" 12 seconds ago Up 10 seconds 0.0.0.0:5432->5432/tcp postgresql-baeldung
As we can see from the output of the above command, the postgresql-baeldung is running with the containerId “bce34bb3c617”. Let’s now explore the docker logs command to monitor the logs:
$ docker logs bce34bb3c617
2022-05-16 18:13:58.868 UTC [1] LOG: starting PostgreSQL 14.2 (Debian 14.2-1.pgdg110+1)
on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
2022-05-16 18:13:58.869 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
2022-05-16 18:13:58.869 UTC [1] LOG: listening on IPv6 address "::", port 5432
Here, the logs contain data of the output stream with the timestamp. The above command doesn’t contain the continuous log output. To view the continuous log output of a container, we need to use the “–follow” option in the docker logs command.
The “–follow” option is one of the most useful Docker options as it allows us to monitor the live logs of a container:
$ docker logs --follow bce34bb3c617
2022-05-16 18:13:58.868 UTC [1] LOG: starting PostgreSQL 14.2 (Debian 14.2-1.pgdg110+1)
on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
...
2022-05-16 18:13:59.018 UTC [1] LOG: database system is ready to accept connections
One of the drawbacks of the above command is that it will contain all the logs from the start. Let’s check out the command to view the continuous log output with the recent records:
$ docker logs --follow --tail 1 bce34bb3c617
2022-05-16 18:13:59.018 UTC [1] LOG: database system is ready to accept connections
We can also use the “since” option with the docker log command to view the file from a particular time:
$ docker logs --since 2022-05-16 bce34bb3c617
2022-05-16 18:13:58.868 UTC [1] LOG: starting PostgreSQL 14.2 (Debian 14.2-1.pgdg110+1)
on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
...
2022-05-16 18:13:59.018 UTC [1] LOG: database system is ready to accept connections
Alternatively, we can also use the docker container logs command instead of the docker logs command:
$ docker container logs --since 2022-05-16 bce34bb3c617
2022-05-16 18:13:58.868 UTC [1] LOG: starting PostgreSQL 14.2 (Debian 14.2-1.pgdg110+1)
on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
...
2022-05-16 18:13:59.018 UTC [1] LOG: database system is ready to accept connections
Here, we can see from the above output that both the command works exactly the same. The docker container logs command is deprecated in the newer versions.
4. Using Default Log File
The Docker stores all the STDOUT and STDERR output in JSON format. Additionally, it is possible to monitor all the live Docker logs from the host machine. By default, Docker stores log files in a dedicated directory on the host using the json-file log driver. The log file directory is /var/lib/docker/containers/<container_id> on the host where the container is running.
To demonstrate, let’s check out the log file of our postgress-baeldung container:
$ cat /var/lib/docker/containers/bce34bb3c6175fe92c50d6e5c8d2045062c2b502b9593a258ceb6cafc9a2356a/
bce34bb3c6175fe92c50d6e5c8d2045062c2b502b9593a258ceb6cafc9a2356a-json.log
{"log":"\r\n","stream":"stdout","time":"2022-05-16T18:13:58.833312658Z"}
{"log":"PostgreSQL Database directory appears to contain a database; Skipping initialization\r\n","stream":"stdout","time":"2022-05-16T18:13:58.833360038Z"}
{"log":"\r\n","stream":"stdout","time":"2022-05-16T18:13:58.833368499Z"}
In the above output, we can see that the data is in JSON format.
5. Clearing the Log File
Sometimes we run out of disk space on our system, and we notice that the Docker log files are taking up a lot of space. For that, we first need to locate the log files and then delete them. Also, make sure that clearing the log file does not impact the state of the running container.
Below is the command to clear all the log files stored on the host machine:
$ truncate -s 0 /var/lib/docker/containers/*/*-json.log
Notice that the above command will not delete the log file. Instead, it will remove all the content inside the log file. By executing the below command, we can delete the log files associated with a specific container:
$ truncate -s 0 /var/lib/docker/containers/dd207f11ebf083f97355be1ae18420427dd2e80b061a7bf6fb0afc326ad04b10/*-json.log
At the start of the container, we can also limit the size of a log file externally using the “–log-opt max-size” and “–log-opt max-file” options of the docker run command:
$ docker run --log-opt max-size=1k --log-opt max-file=5 -itd -e POSTGRES_USER=baeldung -e POSTGRES_PASSWORD=baeldung -p 5432:5432
-v /data:/var/lib/postgresql/data --name postgresql-baeldung postgres
3eec82654fe6c6ffa579752cc9d1fa034dc34b5533b8672ebe7778449726da32
Now, let’s checkout the number of log files and log file size in the /var/lib/docker/containers/3eec82654fe6c6ffa579752cc9d1fa034dc34b5533b8672ebe7778449726da32 directory:
$ ls -la
total 68
drwx------. 4 root root 4096 May 17 02:06 .
drwx------. 5 root root 222 May 17 02:07 ..
drwx------. 2 root root 6 May 17 02:02 checkpoints
-rw-------. 1 root root 3144 May 17 02:02 config.v2.json
-rw-r-----. 1 root root 587 May 17 02:06 3eec82654fe6c6ffa579752cc9d1fa034dc34b5533b8672ebe7778449726da32-json.log
-rw-r-----. 1 root root 1022 May 17 02:06 3eec82654fe6c6ffa579752cc9d1fa034dc34b5533b8672ebe7778449726da32-json.log.1
-rw-r-----. 1 root root 1061 May 17 02:06 3eec82654fe6c6ffa579752cc9d1fa034dc34b5533b8672ebe7778449726da32-json.log.2
-rw-r-----. 1 root root 1056 May 17 02:06 3eec82654fe6c6ffa579752cc9d1fa034dc34b5533b8672ebe7778449726da32-json.log.3
-rw-r-----. 1 root root 1058 May 17 02:06 3eec82654fe6c6ffa579752cc9d1fa034dc34b5533b8672ebe7778449726da32-json.log.4
-rw-r--r--. 1 root root 1501 May 17 02:02 hostconfig.json
-rw-r--r--. 1 root root 13 May 17 02:02 hostname
-rw-r--r--. 1 root root 174 May 17 02:02 hosts
drwx------. 2 root root 6 May 17 02:02 mounts
-rw-r--r--. 1 root root 69 May 17 02:02 resolv.conf
-rw-r--r--. 1 root root 71 May 17 02:02 resolv.conf.hash
Here, we can see that five log files are created, and the size of each log file is 1 kb max. If we delete some log files, in that case, we will generate a new log with the same log file name.
We can also provide the configuration of log max-size and max-file in the /etc/docker/daemon.json file. Let’s look into the configuration of the daemon.json file:
{
"log-driver": "json-file",
"log-opts": {
"max-size": "1k",
"max-file": "5"
}
}
Here, we provided the same configuration in the daemon.json, and importantly, all the new containers will run with this configuration. After updating the daemon.json file, we’ll need to restart the Docker service.
6. Redirecting the Docker Container Logs to a Single File
By default, the Docker containers log files are stored in /var/lib/docker/containers/
To illustrate, let’s look into the command to redirect the container’s log:
$ docker logs -f containername &> baeldung-postgress.log &
Here, in the above command, we redirect all the live logs to the baeldung-postgress.log file. Also, we run this command in the background using the &, so it will keep running until it is explicitly stopped.
7. Conclusion
In this tutorial, we learned different ways to monitor the logs of a container. First, we looked at the docker logs, and the docker container logs command to monitor the live logs. Later, we monitored the logs using the default containers log file.
Finally, we looked into the clearing and redirecting of the log file. In short, we looked into monitoring and truncating the log file.