Lorixen.
EN

Disk space alerts: catch a filling server before it fails

Short answer: A full disk takes an app down in confusing ways: the database stops writing, uploads fail, logins break and error messages often don't mention disk space at all. Set alerts on both free space and free inodes, at two levels such as a warning at 80% used and an urgent alert at 90%, and also alert on how fast usage is growing. Then deal with the usual causes in advance: logs, old backups, container images and temporary files.

Why full disks cause such odd failures

When a server runs out of space, the app rarely says "disk full". Instead you see things like:

  • The database refuses writes or crashes, so orders and sign-ups fail.
  • Sessions can't be saved, so customers are logged out or can't log in.
  • File uploads fail with generic errors.
  • Log writes fail, so the one record that would explain the problem is missing.
  • Deploys fail halfway because the new version can't be unpacked.

Because the symptoms point elsewhere, teams lose time debugging the app before anyone runs df. An alert that fires before the disk fills turns an outage into a routine cleanup.

Watch space and inodes

The standard df command shows space per file system, and df -h prints it in readable sizes. Its -i option lists inode information instead of block usage, and that second view matters.

Every file uses an inode. A server can have plenty of free gigabytes and still be unable to create files because it has run out of inodes, usually because something wrote millions of tiny files, such as session files, cache fragments or a mail queue. Alert on both:

Check Command Typical trigger
Space used df -h Warning at 80%, urgent at 90%
Inodes used df -i Warning at 80%, urgent at 90%
Growth rate Your monitoring tool Full within a few days at the current rate

The thresholds are starting points, not rules. On a large disk, 90% might still leave plenty; on a small one, 80% might be a day from trouble. Adjust after watching real usage.

Alert on the trend, not just the level

A disk at 70% that has been flat for months is fine. A disk at 60% that grew 5% overnight is not. Most monitoring tools can forecast when a disk will fill based on recent growth. A "full within 48 hours" alert catches runaway logs or a stuck job far earlier than a fixed threshold.

Send the urgent alert somewhere a person will see it out of hours, and the warning somewhere it will be read the next working day.

Know what usually fills the disk

On most small business servers, the culprits are predictable.

Logs

Application and system logs grow forever unless something trims them. On Linux, logrotate handles rotation, compression and removal of log files, daily, weekly, monthly or when they grow too large, according to its manual. Check that your app's own log files are covered, not just the system's.

If your server uses systemd's journal, its configuration sets limits. The journald.conf documentation explains that SystemMaxUse= caps how much disk the journal may use and SystemKeepFree= how much it leaves free, with defaults of 10% and 15% of the file system, each capped at 4G. Set explicit values if logs matter on that server.

Backups and database dumps

Local backups that are never deleted are a classic cause. Keep a fixed number, move them off the server, and have the backup job delete old copies only after a new one succeeds.

Container images and build cache

On servers running Docker, old images, stopped containers and build cache pile up with every deploy. Docker's docker system prune removes unused containers, networks, images and build cache, and optionally anonymous volumes. Be careful with volumes: they can hold real data. Run it on a schedule only once you understand what it will remove, and never add the volumes option blindly.

Temporary and upload files

Image processing, exports and failed uploads often leave files in temporary folders. Make sure your app deletes them, and add a scheduled cleanup for anything older than a set age.

Find what's using space quickly

When an alert fires, a short routine saves time:

  1. df -h and df -i to see which file system is affected.
  2. du -sh /var/log /var/lib/docker /tmp /home or similar to find the large folders.
  3. Drill into the largest with du -sh * inside it.
  4. Check for deleted files still held open by a running process, which keep using space until that process restarts.

Write down what you find. If the same cause appears twice, automate the cleanup.

Leave yourself headroom

Some things need free space to work: database maintenance, large migrations, deploys and backups. If your disk is routinely at 85%, you can't run them safely. On cloud servers, growing a disk is usually quick, so it's often cheaper to add space than to keep firefighting. Check your provider's current prices.

A worked example

Take a hypothetical Leeds dental-booking app on one cloud server. One morning nobody could log in, and the errors mentioned session storage. The disk had filled with uncompressed app logs and daily database dumps that were never deleted. The team added the app logs to logrotate, kept seven dumps and copied them to object storage, set journald limits, and added alerts for space, inodes and a fill-time forecast. Three months later the forecast alert caught a debug log level left on after a fix.

FAQ

Is this a problem on managed hosting too?

Yes, although it shows up differently. Managed databases and platforms have storage limits, and many offer their own usage alerts. Switch them on.

What about the database's own disk?

Monitor it separately if the database runs on its own server or volume. Databases often need extra free space for maintenance.

Should alerts go to email?

Warnings can. Urgent alerts should go somewhere people notice quickly, such as a phone notification or team chat.

Can I just set logs to delete after a day?

You can, but you'll lose history you might need for debugging. Keep a sensible window and ship logs elsewhere if you need longer.

If your server has filled up before, or you're not sure anyone would notice, ask us to set up proper disk alerts.

Sources

WhatsAppBook a call