Linux disk full: how to find what is using the space
In short
When a Linux disk is full, df -h shows which filesystem is affected and du -xh --max-depth=1 or ncdu -x show which directory holds the data. If du finds much less than df reports, a running process still holds deleted files open; lsof +L1 lists them, and restarting that service releases the space.
When a Linux server reports No space left on device, three commands answer almost every case: df -h tells you which filesystem is full, du -xh --max-depth=1 tells you which directory holds the data, and lsof +L1 finds space that is still occupied by files that were already deleted. Work in that order and you usually have the cause within five minutes.
Step 1: Which filesystem is full?
A server rarely has just one filesystem. Start with an overview:
df -hTFilesystem Type Size Used Avail Use% Mounted on
/dev/sda1 ext4 79G 75G 0 100% /
/dev/sdb1 ext4 196G 88G 99G 48% /srv/backup
tmpfs tmpfs 3.9G 1.1M 3.9G 1% /runRead the line whose Mounted on column contains the path that failed. In the example the root filesystem is full; the backup disk is irrelevant. Note the detail in the first line: used plus available is less than the size. On ext4, about five percent of the blocks are reserved for root by default, so services running as ordinary users see a full disk while root can still write a little.
If df -h shows free space and writes still fail, check the inodes with df -i. A filesystem can run out of file entries long before it runs out of bytes.
Step 2: Which directory holds the data?
du adds up the size of directory trees. Two options matter: -x stays on one filesystem, so mounted backup disks or network shares are not counted, and --max-depth=1 prints one line per subdirectory.
sudo du -xh --max-depth=1 / 2>/dev/null | sort -h | tail -81.2G /home
2.9G /usr
4.1G /opt
61G /var
75G /Repeat the command for the largest directory until you reach the cause:
sudo du -xh --max-depth=1 /var | sort -h | tail -5
sudo du -xh --max-depth=1 /var/log | sort -h | tail -5If you may install a package, ncdu does the same interactively and is faster to navigate:
sudo apt install ncdu # RHEL family: dnf install ncdu (EPEL repository)
sudo ncdu -x /Move with the arrow keys, press d to delete and q to quit. The delete key in ncdu removes files immediately; treat it with the same care as rm.
To look for single large files instead of directories:
sudo find / -xdev -type f -size +500M -exec ls -lh {} + 2>/dev/nullStep 3: Why do df and du disagree?
Sometimes df reports 75 GB used and du finds only 30 GB. The classic reason: someone deleted a large log file with rm while the service was still writing to it. The directory entry is gone, so du no longer sees the file, but the data blocks stay allocated until the process closes the file.
sudo lsof +L1COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
apache2 1432 www-data 7w REG 8,1 42949672960 0 262811 /var/log/apache2/access.log (deleted)NLINK 0 means the file has no name any more, SIZE/OFF is its size in bytes: here 40 GB. lsof comes from the package of the same name; without it, this works too:
sudo find /proc/*/fd -lname '*(deleted)' -printf '%p -> %l\n' 2>/dev/nullThe clean solution is to restart or reload the service that holds the file (systemctl restart apache2, or systemctl reload if the service reopens its logs on reload). The space is free the moment the process closes the file.
A second cause for a mismatch: data stored underneath a mount point. If /srv/backup was not mounted for a night and the backup script wrote into the empty directory on the root filesystem, those files are hidden as soon as the disk is mounted again. You can look underneath without unmounting:
sudo mkdir /mnt/rootonly
sudo mount --bind / /mnt/rootonly
sudo du -xh --max-depth=1 /mnt/rootonly/srv | sort -h
sudo umount /mnt/rootonlyWhere does the space usually go?
On web and application servers a handful of places account for most full disks:
| Path | Typical cause | Safe way to free space |
|---|---|---|
/var/log | debug logging left on, missing rotation | fix rotation, remove old rotated logs |
/var/log/journal | systemd journal without size limit | journalctl --vacuum-size=500M |
/var/cache/apt | downloaded packages | apt-get clean |
/var/lib/mysql | binary logs, large tables | PURGE BINARY LOGS in the database, never rm |
/var/lib/docker | old images and build cache | docker system df, then docker image prune |
/boot, /usr/lib/modules | old kernels | apt-get autoremove --purge |
/tmp, /var/tmp | forgotten dumps and archives | delete after checking what they are |
/home, /root | database dumps, copies of the web root | move elsewhere or delete |
How do I free space safely?
Start with what cannot hurt:
sudo apt-get clean
sudo journalctl --disk-usage
sudo journalctl --vacuum-size=500M
sudo apt-get autoremove --purgeOn the RHEL family, the equivalents are dnf clean all and dnf autoremove.
For a log file that a service is still writing to, do not use rm. Empty it in place instead:
sudo truncate -s 0 /var/log/myapp/debug.logWarning:
truncate -s 0discards the content irrevocably. If you may need the log to find out why it grew, look at the last lines withtail -n 200or copy a sample to another filesystem first. Never truncate or delete files under/var/lib/mysql,/var/lib/postgresqlor a mail spool; that destroys data.
Then fix the cause, not only the symptom: switch off the debug level, add a logrotate rule, set SystemMaxUse= in /etc/systemd/journald.conf, or move backups to their own filesystem.
Common mistakes
- Deleting an open log with
rm. The space is not released, and now the file is invisible as well. Truncate it or reload the service. - Running
duwithout-x. The result then includes other filesystems,/procand network mounts, and points you to the wrong directory. - Looking only at
/. A separate/var,/tmpor/bootpartition can be full while the root filesystem has plenty of room. - Removing database files by hand. MySQL binary logs or InnoDB files must be removed through the database itself.
- Stopping after cleaning up. If you do not know why the disk filled, it will fill again, usually at night.
- Forgetting dependent damage. After a full disk, check that the database, the mail queue and session storage came back properly; a full disk often leaves truncated files behind.
Checklist
df -hT: which filesystem is at 100 percent?df -i: are the inodes exhausted instead of the blocks?du -xh --max-depth=1orncdu -x: which directory is the largest?lsof +L1: are deleted files still held open?- Free safe space first: package cache, journal, old kernels.
- Truncate active logs instead of deleting them, after looking at their content.
- Remove the cause: log level, rotation, journal limit, backup target.
- Check services that could not write in the meantime.
How Lynx Heartbeat helps
A full disk is one of the few outages that announce themselves days in advance. The Lynx Heartbeat agent measures the fill level and the inode usage of every mount point each minute, and Lynx calculates a forecast from the trend, for example "disk /var full in ≈ 9 days". You get a warning while there is still time to clean up during working hours, not when the database has already stopped. How alerts stay useful instead of noisy is described in alert fatigue, quiet hours and escalation; if a server feels slow rather than full, continue with Linux load average explained. An overview of all measurements is on the features page.
Frequently asked questions
How do I find out what is filling up my Linux disk?
Run df -h to see which filesystem is full, then du -xh --max-depth=1 on its mount point, sorted with sort -h, and descend into the largest directory. ncdu -x does the same interactively.
Why does df show the disk as full while du finds far less data?
Usually a process still holds a deleted file open, typically a log file. The space is only released when the process closes the file. lsof +L1 lists such files; restarting or reloading the service frees the space.
Which files can I delete safely when the disk is full?
The package cache (apt-get clean), old journal entries (journalctl --vacuum-size), rotated and compressed logs, and old kernels via apt-get autoremove are usually safe. Never delete database files, mail spools or files of running services by hand.
Why is the disk full although df shows free gigabytes?
Either the filesystem has run out of inodes, which df -i shows, or the remaining blocks are reserved for root. On ext4 about five percent is reserved by default, so ordinary users hit the limit before root does.