After constantly battling with my system over free space at a crucial time when deadlines had to be met, I finally decided to spend time figuring out what the heck was going on. docker-compose up was failing with errors like INTERNAL ERROR: cannot create temporary directory! and on checking, /root partition indeed had 0% free space. As a stopgap, I deleted unused kernel images, jdks, apt cache, etc. just so I could continue working.
I was surprised to find /var/lib/docker/aufs/diff take 52GB of my precious SSD space. Surprised, because I have a cron job run everyday to delete dangling/stale containers and images, and I expected any disk space used by them to be freed up.
Run du -shc /var/lib/docker to see your damage. Note that this includes files that are used by some of your containers. Our servers also have the same issue, with the directory swelling up to 24GB+. The only difference on the server is it uses overlay2 instead of aufs as the storage driver.
Solution
docker system prune [docs]
Follow options that make sense in your case.
Check your docker version to ensure this command is available. If this doesn't seem to free enough disk space (as seen in my case), upgrade to 17.06.2-ce+ [info] and run the above command again. This seems to fix the issue properly.
Note that upgrading docker will often affect your containers so take care doing this in production!
Related:
https://github.com/moby/moby/issues/21925
https://github.com/moby/moby/issues/22207
Sidenote:
Having a couple of extra linux kernel images installed or Oracle Java installed always helps in freeing up disk space when you really need it!