1. Introduction
Docker is an essential tool for containerization, enabling us to create, deploy, and manage applications in lightweight, portable containers. However, managing Docker images can sometimes present challenges. One common issue is the inability to delete Docker images, even when they appear in the list of images.
In this tutorial, we’ll investigate why this issue occurs and provide practical solutions to resolve it. First, we’ll cover initial troubleshooting steps to set ourselves on the right solution path. Then, we’ll discuss solutions and preventive measures to help us manage our Docker environment more effectively. Let’s get started!
2. Understanding the Problem
It can be frustrating to encounter a Docker image that we cannot delete. This issue typically manifests when we try to remove an image using docker rmi but receive error messages indicating that the image doesn’t exist, even though it’s listed in the output of docker images.
Let’s consider an example scenario to illustrate the problem.
Suppose we have two images listed by docker images -a:
$ docker images -a
REPOSITORY TAG IMAGE ID CREATED SIZE
gcc 7.1.0 855a4f4d1cd9 2 months ago 1.64GB
hello-world latest 1815c82652c0 3 months ago 1.84kB
As we can see, we currently have two images in our repository.
However, attempting to remove these images using the docker rmi command results in an error:
$ docker rmi -f gcc:7.1.0
Error response from daemon: No such image: gcc:7.1.0
$ docker rmi -f 855a4f4d1cd9
Error response from daemon: reference does not exist
$ docker rmi -f hello-world:latest
Error response from daemon: No such image: hello-world:latest
$ docker rmi -f 1815c82652c0
Error response from daemon: reference does not exist
Generally, here is what the process looks like:
- Running docker images -a lists all images in the repository
- Attempting to delete these images with docker rmi [IMAGE_ID] or docker rmi [IMAGE_TAG] results in errors such as “Error response from daemon: No such image: [IMAGE_TAG]” or “Error response from daemon: reference does not exist“
There are several reasons why this issue might occur.
First, we might have a corrupted Docker state. The Docker daemon’s state might be corrupted, leading to inconsistencies between the image list and the actual images. Also, we might have images with dangling or broken references, meaning the references to the image exist, but the actual image data does not.
Lastly, problems with the Docker daemon itself can prevent it from accurately reflecting the state of images.
3. Initial Troubleshooting Steps
Before we dive into solutions, let’s perform some initial troubleshooting steps to rule out common issues and make sure everything is okay initially on our system’s end.
3.1. Checking the Docker Version
First, let’s verify the Docker version we’re working with:
$ docker version
Client:
Version: 20.10.17
...
Experimental: true
Server:
Engine:
Version: 20.10.17
...
Experimental: false
We should ensure that the client and server versions match, as mismatched versions can sometimes lead to unexpected behavior, such as the error we’re encountering.
3.2. Checking the Docker Daemon Status
We should also ensure that the Docker daemon is running correctly:
$ sudo systemctl status docker
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2024-07-17 12:00:00 UTC; 1 day 3h ago
Docs: https://docs.docker.com
Main PID: 1234 (dockerd)
Tasks: 8
Memory: 32.0M
CGroup: /system.slice/docker.service
└─1234 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
Ensuring the Docker daemon is running is crucial because if it is inactive, Docker commands will not function correctly, leading to issues such as being unable to remove images.
However, if the daemon is not running (inactive), we need to start it:
$ sudo systemctl start docker
This starts the Docker daemon, allowing Docker commands to execute properly.
3.3. Ensuring No Containers Are Running
We also need to ensure that no running or stopped containers might be using the images we want to delete.
To do this, we can list all containers:
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e0a1c7b4f4a9 gcc:7.1.0 "/bin/bash" 2 months ago Exited (0) 2 weeks ago nostalgic_pike
f3a3c4d2e2a3 hello-world "/hello" 3 months ago Exited (0) 3 months ago keen_pike
Here, we list all containers, including those that are stopped. Knowing which containers exist helps us identify if any are using the images we want to delete.
However, if any containers are using the images, we need to stop and remove them:
$ docker stop [CONTAINER_ID]
$ docker rm [CONTAINER_ID]
Running or stopped containers can hold references to images, preventing their deletion. Stopping and removing these containers ensures that the images are not in use.
3.4. Verifying Image References
Next, we should verify that the image references are correct by listing all images:
$ docker images -a
REPOSITORY TAG IMAGE ID CREATED SIZE
gcc 7.1.0 855a4f4d1cd9 2 months ago 1.64GB
hello-world latest 1815c82652c0 3 months ago 1.84kB
Then, we inspect the specific images we want to delete, e.g., 855a4f4d1cd9:
$ docker inspect 855a4f4d1cd9
[
{
"Id": "sha256:855a4f4d1cd9b2f0e01a2d2f43203e6bb882fa4e537f4a7b56457ef6df37b4f4",
...
}
]
Inspecting the images provides detailed information about the image’s configuration and status, helping us identify any discrepancies or issues with the image references.
If everything checks out, we can now look into solutions.
4. Common Solutions
If the initial troubleshooting steps didn’t resolve the issue, it’s time for us to explore more common solutions. These steps handle typical problems that might cause Docker images to be undeletable.
Let’s see these possible solutions one after the other.
4.1. Removing Images Using Image IDs and Tags
First, we should try using the force flag (-f) with docker rmi, which often helps to remove stubborn images:
$ docker rmi -f [IMAGE_ID]
$ docker rmi -f [IMAGE_TAG]
Untagged: gcc:7.1.0
Deleted: sha256:855a4f4d1cd9b2f0e01a2d2f43203e6bb882fa4e537f4a7b56457ef6df37b4f4
Here, the -f flag forces the image removal, ignoring any warnings or errors about the image being used by containers. This can resolve issues where the image is stubbornly refusing to be deleted.
4.2. Using docker image prune
Docker also provides built-in commands to clean up unused images.
First, we can use the docker image prune command to remove dangling images:
$ docker image prune
WARNING! This will remove all dangling images.
Are you sure you want to continue? [y/N] y
Deleted Images:
sha256:1815c82652c0
Total reclaimed space: 1.84kB
Further, we can also use docker image prune -a to remove all unused images:
$ docker image prune -a
WARNING! This will remove all images without at least one container associated to them.
Are you sure you want to continue? [y/N] y
Deleted Images:
sha256:855a4f4d1cd9
sha256:1815c82652c0
Total reclaimed space: 1.64GB
These commands can help clean up images that are not associated with any containers, potentially resolving the issue of undeletable images.
4.3. Restarting the Docker Daemon
Sometimes, simply restarting the Docker daemon can resolve discrepancies between the image list and the actual images:
$ sudo systemctl restart docker
This refreshes Docker’s state, potentially resolving issues caused by temporary inconsistencies or bugs in the daemon.
5. Advanced Solutions
If common solutions do not work, let’s consider more advanced methods to resolve the issue.
5.1. Manually Cleaning Up Broken References
Broken references in Docker’s image database can cause the issue. Therefore, we can manually clean up these references by locating and deleting the problematic files. The image database files are typically found in /var/lib/docker/image/devicemapper/imagedb/content/sha256.
First, we need to identify the problematic image ID. To do this, we navigate to the directory where the image database files are stored:
$ cd /var/lib/docker/image/devicemapper/imagedb/content/sha256
$ ls
0f27c67f44eae86eac9d7b43e89a0c6fbf21d256
1815c82652c0
855a4f4d1cd9
...
Then, we locate the file corresponding to the problematic image ID and delete it, e.g., 855a4f4d1cd9:
$ sudo rm -f 855a4f4d1cd9
Here, we remove the specific image file, which can help resolve issues caused by broken references in the Docker image database.
Now, we can restart Docker to apply the changes:
$ sudo systemctl restart docker
Restarting Docker applies the changes and helps resolve any remaining issues related to the removed image.
5.2. Using Docker System Commands
Docker system commands can help clean up and manage Docker resources.
We can use docker system prune to remove all unused data (images, containers, volumes, and networks):
$ docker system prune
WARNING! This will remove:
- all stopped containers
- all networks not used by at least one container
- all dangling images
- all build cache
Are you sure you want to continue? [y/N] y
Deleted Containers:
e0a1c7b4f4a9
Deleted Images:
sha256:855a4f4d1cd9
sha256:1815c82652c0
Total reclaimed space: 1.64GB
With this, we clean up unused Docker resources, which can help resolve issues by removing any lingering references or data.
For a more detailed overview of Docker disk usage:
$ docker system df
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 10 2 10GB 8GB (80%)
Containers 5 1 500MB 400MB (80%)
Local Volumes 3 1 300MB 200MB (66%)
Build Cache 0 0 0B 0B
As we can see, we have a detailed summary of Docker disk usage, helping to identify which resources are consuming space and might need cleanup.
5.3. Clearing the Docker State
If the issue persists, as a last resort, we need to clear and reset Docker’s entire state. This involves stopping the Docker service, removing the Docker directory, and then restarting the service.
However, we should be aware that this will remove all Docker data as we’ll need to pull or rebuild all our images and recreate our containers, so let’s proceed with caution:
$ sudo systemctl stop docker
$ sudo rm -rf /var/lib/docker
$ sudo systemctl start docker
Sequentially, we stop the Docker service, remove all Docker data, and then restart the service. This will start Docker fresh without any existing data. After this, we should try pulling and deleting the images again to see if the problem is resolved.
6. Docker Storage Drivers
Docker uses storage drivers to manage the contents of images and containers. The choice of storage driver can significantly affect performance and, in some cases, the behavior of image-related operations. Common storage drivers include overlay2, devicemapper, aufs, btrfs, and zfs. Each driver has its own way of implementing layered storage for Docker images.
In rare cases, the storage driver in use can sometimes contribute to issues with image deletion. For instance:
- Layer dependencies: Some drivers maintain complex layer dependencies that can prevent image deletion if not properly resolved
- Caching mechanisms: Certain drivers use caching to improve performance, which might lead to stale data and image reference issues
- File system peculiarities: Drivers that interact directly with specific file systems (like btrfs or zfs) may encounter unique challenges related to those file systems
If we suspect our storage driver might be contributing to image deletion issues, we can consider changing it.
To do this, first, we check which storage driver we’re currently using with docker info:
$ docker info | grep "Storage Driver"
Storage Driver: overlay2
Then, we stop the Docker daemon:
$ sudo systemctl stop docker
Next, we edit the Docker daemon configuration file (/etc/docker/daemon.json) with any preferred editor like vi:
$ sudo vi /etc/docker/daemon.json
# Editing the /etc/docker/daemon.json file
{
"storage-driver": "overlay2"
}
Notably, changing storage drivers will make existing images and containers inaccessible. Therefore, we can back up important data before proceeding.
Finally, we restart Docker:
$ sudo systemctl start docker
This process should finally fix our issue if it relates to the storage driver.
7. Preventive Measures
To avoid encountering similar issues in the future, we should consider implementing preventive measures like scheduling regular cleanups to remove unused images and containers. We can automate this process with cron jobs.
For example, let’s create a cron job to run docker image prune -a weekly:
$ crontab -e
...
0 0 * * SUN docker image prune -a -f
Here, we schedule the cleanup every Sunday at midnight.
Further, we can also use monitoring tools to keep an eye on Docker’s performance and resource usage. We can leverage tools like Prometheus, Grafana, and Docker’s built-in monitoring features to help us track and manage Docker resources effectively.
8. Conclusion
In this article, we’ve explored the perplexing issue of Docker images appearing in listings but resisting deletion attempts. We’ve also walked through a comprehensive troubleshooting process, starting with basic verification steps and moving on to advanced solutions like manually cleaning Docker image references.
We’ve learned that this problem can stem from various sources, including corrupted Docker states, interrupted operations, or file system issues.