Skip to content

Linux disk full: how to find what is using the space

When a Linux disk is full, df -h shows which filesystem is affected and du -xh --max-depth=1 or ncdu -x show which directory holds the data. If du finds much less than df reports, a running process still holds deleted files open; lsof +L1 lists them, and restarting that service releases the space.

When a Linux server reports No space left on device, three commands answer almost every case: df -h tells you which filesystem is full, du -xh --max-depth=1 tells you which directory holds the data, and lsof +L1 finds space that is still occupied by files that were already deleted. Work in that order and you usually have the cause within five minutes.

Step 1: Which filesystem is full?

A server rarely has just one filesystem. Start with an overview:

df -hT
Filesystem     Type   Size  Used Avail Use% Mounted on
/dev/sda1      ext4    79G   75G     0 100% /
/dev/sdb1      ext4   196G   88G   99G  48% /srv/backup
tmpfs          tmpfs  3.9G  1.1M  3.9G   1% /run

Read the line whose Mounted on column contains the path that failed. In the example the root filesystem is full; the backup disk is irrelevant. Note the detail in the first line: used plus available is less than the size. On ext4, about five percent of the blocks are reserved for root by default, so services running as ordinary users see a full disk while root can still write a little.

If df -h shows free space and writes still fail, check the inodes with df -i. A filesystem can run out of file entries long before it runs out of bytes.

Step 2: Which directory holds the data?

du adds up the size of directory trees. Two options matter: -x stays on one filesystem, so mounted backup disks or network shares are not counted, and --max-depth=1 prints one line per subdirectory.

sudo du -xh --max-depth=1 / 2>/dev/null | sort -h | tail -8
1.2G    /home
2.9G    /usr
4.1G    /opt
61G     /var
75G     /

Repeat the command for the largest directory until you reach the cause:

sudo du -xh --max-depth=1 /var | sort -h | tail -5
sudo du -xh --max-depth=1 /var/log | sort -h | tail -5

If you may install a package, ncdu does the same interactively and is faster to navigate:

sudo apt install ncdu        # RHEL family: dnf install ncdu (EPEL repository)
sudo ncdu -x /

Move with the arrow keys, press d to delete and q to quit. The delete key in ncdu removes files immediately; treat it with the same care as rm.

To look for single large files instead of directories:

sudo find / -xdev -type f -size +500M -exec ls -lh {} + 2>/dev/null

Step 3: Why do df and du disagree?

Sometimes df reports 75 GB used and du finds only 30 GB. The classic reason: someone deleted a large log file with rm while the service was still writing to it. The directory entry is gone, so du no longer sees the file, but the data blocks stay allocated until the process closes the file.

sudo lsof +L1
COMMAND   PID     USER   FD   TYPE DEVICE    SIZE/OFF NLINK    NODE NAME
apache2  1432 www-data    7w   REG    8,1 42949672960     0  262811 /var/log/apache2/access.log (deleted)

NLINK 0 means the file has no name any more, SIZE/OFF is its size in bytes: here 40 GB. lsof comes from the package of the same name; without it, this works too:

sudo find /proc/*/fd -lname '*(deleted)' -printf '%p -> %l\n' 2>/dev/null

The clean solution is to restart or reload the service that holds the file (systemctl restart apache2, or systemctl reload if the service reopens its logs on reload). The space is free the moment the process closes the file.

A second cause for a mismatch: data stored underneath a mount point. If /srv/backup was not mounted for a night and the backup script wrote into the empty directory on the root filesystem, those files are hidden as soon as the disk is mounted again. You can look underneath without unmounting:

sudo mkdir /mnt/rootonly
sudo mount --bind / /mnt/rootonly
sudo du -xh --max-depth=1 /mnt/rootonly/srv | sort -h
sudo umount /mnt/rootonly

Where does the space usually go?

On web and application servers a handful of places account for most full disks:

PathTypical causeSafe way to free space
/var/logdebug logging left on, missing rotationfix rotation, remove old rotated logs
/var/log/journalsystemd journal without size limitjournalctl --vacuum-size=500M
/var/cache/aptdownloaded packagesapt-get clean
/var/lib/mysqlbinary logs, large tablesPURGE BINARY LOGS in the database, never rm
/var/lib/dockerold images and build cachedocker system df, then docker image prune
/boot, /usr/lib/modulesold kernelsapt-get autoremove --purge
/tmp, /var/tmpforgotten dumps and archivesdelete after checking what they are
/home, /rootdatabase dumps, copies of the web rootmove elsewhere or delete

How do I free space safely?

Start with what cannot hurt:

sudo apt-get clean
sudo journalctl --disk-usage
sudo journalctl --vacuum-size=500M
sudo apt-get autoremove --purge

On the RHEL family, the equivalents are dnf clean all and dnf autoremove.

For a log file that a service is still writing to, do not use rm. Empty it in place instead:

sudo truncate -s 0 /var/log/myapp/debug.log

Warning: truncate -s 0 discards the content irrevocably. If you may need the log to find out why it grew, look at the last lines with tail -n 200 or copy a sample to another filesystem first. Never truncate or delete files under /var/lib/mysql, /var/lib/postgresql or a mail spool; that destroys data.

Then fix the cause, not only the symptom: switch off the debug level, add a logrotate rule, set SystemMaxUse= in /etc/systemd/journald.conf, or move backups to their own filesystem.

Common mistakes

  • Deleting an open log with rm. The space is not released, and now the file is invisible as well. Truncate it or reload the service.
  • Running du without -x. The result then includes other filesystems, /proc and network mounts, and points you to the wrong directory.
  • Looking only at /. A separate /var, /tmp or /boot partition can be full while the root filesystem has plenty of room.
  • Removing database files by hand. MySQL binary logs or InnoDB files must be removed through the database itself.
  • Stopping after cleaning up. If you do not know why the disk filled, it will fill again, usually at night.
  • Forgetting dependent damage. After a full disk, check that the database, the mail queue and session storage came back properly; a full disk often leaves truncated files behind.

Checklist

  1. df -hT: which filesystem is at 100 percent?
  2. df -i: are the inodes exhausted instead of the blocks?
  3. du -xh --max-depth=1 or ncdu -x: which directory is the largest?
  4. lsof +L1: are deleted files still held open?
  5. Free safe space first: package cache, journal, old kernels.
  6. Truncate active logs instead of deleting them, after looking at their content.
  7. Remove the cause: log level, rotation, journal limit, backup target.
  8. Check services that could not write in the meantime.

How Lynx Heartbeat helps

A full disk is one of the few outages that announce themselves days in advance. The Lynx Heartbeat agent measures the fill level and the inode usage of every mount point each minute, and Lynx calculates a forecast from the trend, for example "disk /var full in ≈ 9 days". You get a warning while there is still time to clean up during working hours, not when the database has already stopped. How alerts stay useful instead of noisy is described in alert fatigue, quiet hours and escalation; if a server feels slow rather than full, continue with Linux load average explained. An overview of all measurements is on the features page.

Frequently asked questions

How do I find out what is filling up my Linux disk?

Run df -h to see which filesystem is full, then du -xh --max-depth=1 on its mount point, sorted with sort -h, and descend into the largest directory. ncdu -x does the same interactively.

Why does df show the disk as full while du finds far less data?

Usually a process still holds a deleted file open, typically a log file. The space is only released when the process closes the file. lsof +L1 lists such files; restarting or reloading the service frees the space.

Which files can I delete safely when the disk is full?

The package cache (apt-get clean), old journal entries (journalctl --vacuum-size), rotated and compressed logs, and old kernels via apt-get autoremove are usually safe. Never delete database files, mail spools or files of running services by hand.

Why is the disk full although df shows free gigabytes?

Either the filesystem has run out of inodes, which df -i shows, or the remaining blocks are reserved for root. On ext4 about five percent is reserved by default, so ordinary users hit the limit before root does.

Servers

Related articles

All articles