Docker was hiding half a terabyte on our CI runner

A few weeks ago one of our self-hosted CI runners started failing builds with no space left on device. The machine has a 700GB disk, and apart from Docker, nothing on it should come anywhere close to filling that.

I started with the obvious fix and ran docker system prune. It reported 254GB reclaimed and I felt very pleased with myself, but two weeks later the builds were failing again.

Most of what that prune freed was build cache, which on a CI runner grows straight back. The real problem was another 238GB that the prune never touched. Together that's half a terabyte (492GB, to be exact) that Docker had been holding on to without needing it. It was spread across six months of old images from every PR, over a thousand orphaned volumes, databases from projects nobody remembered, and log files that no command cleans up.

The sections below follow the order you should look in. Each one gives the command, what it printed on our runner, what it actually deletes, and whether it can lose you data.

Short version: docker system prune only removes the easy stuff. Docker keeps disk space in six places: stopped containers, images, build cache, networks, anonymous volumes and named volumes. Container log files are a seventh, and no prune command touches them. Each one needs handling separately and in the right order, and two of them can hold data you can't get back.

Step 0: find out where the space went

Don't delete anything until you've measured it. Docker will tell you what it's using:

docker system df

This was our runner before the cleanup:

TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          214       9         148.3GB   131.2GB (88%)
Containers      163       9         19.4GB    19.1GB (98%)
Local Volumes   1187      12        111.2GB   108.5GB (97%)
Build Cache     3412      0         212.7GB   212.7GB

That adds up to 492GB, of which Docker reckons 472GB is reclaimable. 1,187 volumes on a CI runner is a warning sign, and we'll come back to it.

For a breakdown by item, showing each image, container and volume and how big it is, add -v:

docker system df -v

Images space usage:

REPOSITORY   TAG       IMAGE ID       CREATED        SIZE      SHARED SIZE   UNIQUE SIZE   CONTAINERS
myapp        pr-4457   d6fa79e795cf   2 days ago     1.87GB    1.12GB        748.2MB       0
myapp        pr-4450   5ae57564875a   3 days ago     1.87GB    1.12GB        748.1MB       0
myapp        pr-4442   538847bdfead   3 days ago     1.86GB    1.12GB        741.6MB       0
...
node         20        8f693eaa7e0a   6 months ago   1.59GB    0B            1.592GB       0
golang       1.23      60deed95d388   14 months ago  1.24GB    32B           1.241GB       0

Local Volumes space usage:

VOLUME NAME                                                        LINKS     SIZE
23916acdbd51a9f63626a26152f174a05a420749673c517c1f3ac97bfdcd832c   0         61.2MB
b9397c0e36b1d9b520ea6eb27a48fed183d5efd95db9bb2e524c4fe69e7ebeb2   0         58.9MB
billing-v1_pgdata                                                  0         6.2GB
...

It's slow on a machine this cluttered because it has to measure every volume, but it's worth the wait. Save the output somewhere so you can compare afterwards. The CONTAINERS column showing 0 next to a pile of myapp tags was the first clue.

docker system df doesn't count container log files, so check those separately. You'll need root, because they live under Docker's data directory:

sudo du -sh /var/lib/docker/containers/*/*-json.log | sort -h | tail -5

412M    /var/lib/docker/containers/3f2a8a6e0ca0.../3f2a8a6e0ca0...-json.log
1.1G    /var/lib/docker/containers/2f69a6e7b437.../2f69a6e7b437...-json.log
1.9G    /var/lib/docker/containers/08544a3e98bb.../08544a3e98bb...-json.log
2.6G    /var/lib/docker/containers/d7affb84d4f6.../d7affb84d4f6...-json.log
20.4G   /var/lib/docker/containers/24f2c9f8e39a.../24f2c9f8e39a...-json.log

Ours came to another 27GB, all from running containers, and 20GB of it came from one noisy service that had been running for months.

docker system prune skips most of it

With no flags, docker system prune lists what it will remove (this is from Docker 29):

WARNING! This will remove:
  - all stopped containers
  - all networks not used by at least one container
  - all dangling images
  - unused build cache

A lot is missing from that list:

• Dangling images are untagged layers (the <none>:<none> ones). An old myapp:pr-4410 image that nothing runs any more is tagged, so it isn't dangling, and it stays.
• Volumes aren't on the list at all, neither anonymous nor named.
• Log files belong to containers, so a running container keeps its log however big it is.

Adding the flags gets you further:

$ docker system prune --all --volumes
WARNING! This will remove:
  - all stopped containers
  - all networks not used by at least one container
  - all anonymous volumes not used by at least one container
  - all images without at least one container associated to them
  - all build cache

That only says anonymous volumes. Since Docker 23, --volumes no longer removes named volumes, and no flag on docker system prune will. You have to use docker volume prune --all, which is a good thing, because named volumes are where databases keep their data.

So the "one command" is really several, and running the big one blind is a bad idea because you can't see what you're about to lose. The steps below take them one at a time.

Step 1: stopped containers

A stopped container still takes up space, because it keeps its writable layer, which holds every file it created or changed while it ran. A container that wrote a 2GB test fixture to /tmp and then exited still holds that 2GB.

List them first:

docker ps --all --filter status=exited --filter status=created \
  --format 'table {{.ID}}\t{{.Image}}\t{{.Status}}\t{{.Size}}'

CONTAINER ID   IMAGE           STATUS                     SIZE
0bc4c835124b   node:22         Created                    4.1kB (virtual 1.22GB)
6dadd56bb84f   myapp:pr-4457   Exited (0) 2 days ago      214MB (virtual 2.08GB)
890358cacab6   myapp:pr-4450   Exited (1) 3 days ago      2.1GB (virtual 3.97GB)
670cb73a6226   postgres:17     Exited (0) 3 days ago      63.4kB (virtual 646MB)
c9c7af3eac5f   myapp:pr-4442   Exited (0) 3 days ago      187MB (virtual 2.05GB)
...

There were 154 of these. The Created one at the top was never started, but it still counts and still pins its image.

Then remove them, with a filter to keep anything recent:

# only containers that stopped more than a day ago
docker container prune --filter "until=24h"

Deleted Containers:
0bc4c835124b4b1ab3c68c159231d4495df68f267020c80952a7276e9f2ad345
6dadd56bb84f37190a09cdf8a3002cbafd727243095f26f4a4c01fc6a88db915
890358cacab64995ea58acbb6ee549be15885647ec48aee487a2db88cca27378
...

Total reclaimed space: 19.1GB

⚠️ Do this step first. While a stopped container exists, its image and volumes count as "in use", so the later prunes skip them. If you prune volumes before containers, very little happens and it isn't obvious why.

⚠️ Risk: a stopped container may hold the only copy of something, such as logs you wanted for a post-mortem, files someone docker cp'd in, or a container someone meant to docker start again. If you're unsure, docker logs <id> and docker diff <id> show you what's in it.

Step 2: images

There are two kinds of unused image:

1. Dangling images are untagged layers left behind when a tag moves to a newer build. They're always safe to remove.
2. Unused images are tagged, but no container (running or stopped) uses them. This is where most of the space is.

Start with the dangling ones:

docker image prune

Deleted Images:
deleted: sha256:a31352d2fceab10d49783ce2ad1176d30221b6134fed17eae6406cb16385671a
deleted: sha256:2bdeb27d1be32898f370f529163033163663b75f14ca8d91927e62104efb114f
...

Total reclaimed space: 22.3GB

Then list what's tagged:

docker image ls --format 'table {{.Repository}}\t{{.Tag}}\t{{.Size}}'

REPOSITORY   TAG       SIZE
myapp        latest    1.87GB
myapp        pr-4457   1.87GB
myapp        pr-4450   1.87GB
myapp        pr-4442   1.86GB
...
python       3.12      1.62GB
node         24        1.65GB
node         22        1.64GB
node         20        1.59GB
golang       1.23      1.24GB
mongo        7         1.18GB
postgres     17        646MB
postgres     16        642MB

There were old tags of our own app from every PR built in six months, three major versions of the Node base image, two of Postgres, and a Go toolchain nothing had used in over a year. On a CI runner every one of these can be rebuilt or pulled again, so we removed every image that no container uses:

docker image prune --all

Deleted Images:
untagged: myapp:pr-4410
deleted: sha256:a31352d2fceab10d49783ce2ad1176d30221b6134fed17eae6406cb16385671a
untagged: myapp:pr-4417
deleted: sha256:64121dc70079a43f1fe0d19f473bcbe3baae367fa641ff690e4adf1532c7670c
...
untagged: node:20
untagged: golang:1.23
...

Total reclaimed space: 108.9GB

Reclaimed: 131.2GB across both commands.

⚠️ Risk: --all has no idea which images are hard to get back. If an image was built locally and never pushed, or came from a registry that has since deleted it or that you can't reach offline, it's gone for good. On anything other than a throwaway runner, protect those images with a label, and use an age filter to keep recent ones:

docker image prune --all --filter "until=168h" --filter "label!=keep=true"

⚠️ Cost: the next build or docker compose up has to pull everything again. On a CI runner that means one slow build, but on a laptop on hotel wifi it can cost you the afternoon.

Step 3: build cache

BuildKit keeps cached layers for every build step it has run, including cache mounts (RUN --mount=type=cache) that hold package manager caches. Docker doesn't count these as images, so docker image ls won't show them.

docker builder du

ID                           RECLAIMABLE   SIZE      LAST ACCESSED
ml0qdscb7gq0bxqvrpyzut6um    true          1.12GB    2 days ago
4rzhex22ahlvbvupjj0n7pxko    true          748MB     2 days ago
3on6uhg4oa0f48fsqwddpfxn8    true          748MB     3 days ago
...
Shared:         4.81GB
Private:        207.9GB
Reclaimable:    212.7GB
Total:          212.7GB

Then prune it:

docker builder prune

ID                           RECLAIMABLE   SIZE      LAST ACCESSED
ml0qdscb7gq0bxqvrpyzut6um    true          1.12GB    2 days ago
4rzhex22ahlvbvupjj0n7pxko    true          748MB     2 days ago
...
Total:  212.7GB

Reclaimed: 212.7GB. This is the one part docker system prune does deal with, and it's why that first prune looked so impressive. Plain docker builder prune clears all unused cache, and --all only adds BuildKit's own internal images, which came to a few megabytes for us.

The catch is that build cache comes straight back. Every PR build on that runner used slightly different layers, so the cache was back above 100GB within days. Pruning it by hand on a busy builder can't keep up, so give BuildKit a limit and let it keep the cache trimmed:

# keep at most 50GB of build cache, run on a schedule
docker builder prune --force --max-used-space 50GB

(Check docker builder prune --help on your version, because these options have changed name more than once. Older releases used --keep-storage.)

⚠️ Cost: your next builds start with an empty cache.

Step 4: networks

Networks use almost no disk, so they won't fill it, but they cause a different problem. Every docker compose up creates a network, and Docker hands out subnets from a fixed pool, so once enough of them are left lying around you'll start seeing:

could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network

Clean them up while you're here:

docker network ls

NETWORK ID     NAME                 DRIVER    SCOPE
4fe365105a39   billing-v1_default   bridge    local
8cb90811bc3d   bridge               bridge    local
a1d355dc54c9   ci-job-1817          bridge    local
7b6cb443aeaf   ci-job-1822          bridge    local
...
166522cb1220   host                 host      local
e18b8c36e447   none                 null      local
6681b84b967f   old-portal_default   bridge    local
eca88e9a37da   portal_default       bridge    local

docker network prune --filter "until=24h"

Deleted Networks:
search-poc_default
old-portal_default
ci-job-1817
ci-job-1822
...

Reclaimed: 0GB of disk, and 187 networks. This was the step that stopped the random compose failures on our runner.

⚠️ Risk: low. Docker won't remove a network that a container is attached to, and it never removes the built-in bridge, host and none networks. It will remove a network you created by hand and meant to reuse later, though.

Step 5: anonymous volumes

This is where the 1,187 volumes came from, and they're easy to miss.

When an image declares a VOLUME in its Dockerfile, as the official postgres, mysql and mongo images all do, every docker run without a matching -v creates a new anonymous volume with a random 64-character name. When the container goes away the volume stays, unless the container was started with --rm or removed with docker rm -v.

So a test suite that starts a fresh Postgres container for every CI job leaves one orphaned database volume behind per job, every day, for months.

Count them:

# anonymous volumes (Docker 23+ labels them)
docker volume ls --quiet --filter label=com.docker.volume.anonymous | wc -l
# 1141

# volumes no container references
docker volume ls --quiet --filter dangling=true | wc -l
# 1175

Then remove them:

docker volume prune

Deleted Volumes:
ac229af15c128e295cec26e8327aa30a187b873009bcbae5aafdc054540f0a08
8b143909b8f6b3ee96886fb0b84f13dd078fc986f64f783586d98c4c094384d6
e243f062d1440222819084ada30f09ed2652db9a766ca3cb61301bfbb0fefa0e
...

Total reclaimed space: 64GB

Since Docker 23, docker volume prune with no flags removes only anonymous volumes that no container uses. That removed 1,138 of them, and it left the other 37 unused volumes alone because they have names.

Reclaimed: 64.0GB.

To stop making more, add --rm to throwaway docker run commands, use docker compose down -v in CI teardown, and use docker rm -v when removing containers by hand:

docker run --rm -e POSTGRES_PASSWORD=test postgres:17

⚠️ Risk: low, but not zero. If somebody ran a database without -v and has been putting real data in it, that data is in an anonymous volume, and pruning the volume after its container is gone deletes the data. Check before you assume nobody did.

Step 6: named volumes

This is the step where you can lose real data, so slow down.

Named volumes (-v pgdata:/var/lib/postgresql/data, or the volumes: section of a compose file) are where you deliberately keep data. Docker counts a named volume as "unused" when no container references it, and Step 1 removed the stopped containers. If you stopped your local database last week, its volume now counts as unused and looks exactly like junk.

List what's left, with sizes, and read the list properly:

docker system df -v | sed -n '/Local Volumes space usage/,/Build cache usage/p'

Local Volumes space usage:

VOLUME NAME                                                        LINKS     SIZE
billing-v1_pgdata                                                  0         6.2GB
search-poc_pgdata                                                  0         4.8GB
load-test_pgdata                                                   0         9.1GB
billing-v1_redis                                                   0         412MB
old-portal_redis                                                   0         388MB
portal_pgdata                                                      1         2.1GB
load-test_redis                                                    0         1.3GB
search-poc_redis                                                   0         96MB
old-portal_pgdata                                                  0         3.4GB
...

LINKS 0 means no container uses the volume, and that's all Docker knows. It can't tell you whether load-test_pgdata is a forgotten experiment or the only copy of last quarter's benchmark results.

Compose labels the volumes it creates with the project name, so you can see which project owns each one:

docker volume ls --format 'table {{.Name}}\t{{.Label "com.docker.compose.project"}}'

VOLUME NAME          project
billing-v1_pgdata    billing-v1
billing-v1_redis     billing-v1
load-test_pgdata     load-test
load-test_redis      load-test
old-portal_pgdata    old-portal
old-portal_redis     old-portal
portal_pgdata        portal
search-poc_pgdata    search-poc
search-poc_redis     search-poc
...

Back up anything you might want before you remove it:

docker run --rm -v load-test_pgdata:/data -v "$PWD":/backup alpine \
  tar czf /backup/load-test_pgdata.tgz -C /data .

Then remove volumes by name where you can, rather than sweeping them all:

docker volume rm billing-v1_pgdata billing-v1_redis search-poc_pgdata search-poc_redis

billing-v1_pgdata
billing-v1_redis
search-poc_pgdata
search-poc_redis

You can sweep with docker volume prune --all, which removes every named volume that no container uses, but on a shared machine that's how someone's three-week-old local dataset disappears.

Reclaimed: 38.1GB, from 23 volumes belonging to compose projects that no longer exist. We checked every one by name first and kept the other 14.

⚠️ Risk: high. There's no undo and no recycle bin, and volume data isn't stored in any image or registry, so if it isn't backed up, it's gone.

Step 7: container logs

No prune command touches the log of a running container, and Docker's default json-file log driver has no size limit, so a chatty container keeps writing until the disk is full.

To free the space now, truncate the file in place. Don't delete it, because Docker keeps the file open and the space won't come back until the container restarts:

log=$(docker inspect --format '{{.LogPath}}' portal-api)
sudo du -sh "$log"
sudo truncate --size 0 "$log"
sudo du -sh "$log"

20.4G   /var/lib/docker/containers/24f2c9f8e39a.../24f2c9f8e39a...-json.log
0       /var/lib/docker/containers/24f2c9f8e39a.../24f2c9f8e39a...-json.log

Then do the same for every other running container:

for id in $(docker ps --quiet); do
  sudo truncate --size 0 "$(docker inspect --format '{{.LogPath}}' "$id")"
done

To stop it happening again, set a default limit in /etc/docker/daemon.json and restart Docker:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "50m",
    "max-file": "3"
  }
}

Reclaimed: 27.0GB.

⚠️ Gotcha: the new defaults only apply to containers created after the change, and existing containers keep their old, unlimited settings until they're recreated. Truncating also throws away log history, so if anyone might need it, copy it somewhere first.

Where the 492GB was

This is the runner afterwards:

TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          9         9         17.1GB    0B (0%)
Containers      9         9         312.6MB   0B (0%)
Local Volumes   26        12        9.1GB     6.4GB (70%)
Build Cache     0         0         0B        0B

And this is where the half terabyte was hiding:

The plain prune got about half of it, and most of that half was build cache that refilled within days. The 238GB it couldn't see is why the disk kept filling up.

Keeping the runner clean

A one-off clean-up only resets the clock, so on our runners it's now a scheduled job, and a deliberately conservative one:

#!/usr/bin/env bash
set -euo pipefail

docker container prune --force --filter "until=24h"
docker image prune --all --force --filter "until=168h" --filter "label!=keep=true"
docker builder prune --force --max-used-space 50GB
docker network prune --force --filter "until=24h"
docker volume prune --force   # anonymous only; named volumes are never swept

It's still a set of separate commands run in order, and it still never sweeps named volumes. With log limits in daemon.json, --rm on throwaway containers and down -v in CI teardown as well, that runner has stayed at about 35% disk use since.

If you're on Docker Desktop, everything lives inside a virtual machine disk file. Pruning frees space inside the VM, but the file on your host may not shrink straight away, so check Settings → Resources for the disk size and the reclaim options on your version.

What we build

This runner builds and tests Procaptcha and Prosopo Protect. Procaptcha is a drop-in replacement for reCAPTCHA and hCaptcha that is cookieless by default and free for the first 10,000 verifications a month, and Protect stops bots at the edge, before they reach your site. If bots are filling your forms rather than your disks, start free or get in touch.

TL;DR

1. Run docker system df -v first, and measure before you delete.
2. Remove stopped containers first, otherwise everything they reference counts as "in use".
3. Run docker image prune --all with an until filter and a keep label.
4. Run docker builder prune, then give BuildKit a cap so it trims itself.
5. Run docker network prune. It frees no disk, but it fixes subnet exhaustion.
6. Use docker volume prune for anonymous volumes, and stop making them with --rm.
7. For named volumes, list them, back them up and remove them by name. Never sweep them blind.
8. Cap container logs in daemon.json, because no prune will do it for you.

Happy pruning!