Lorixen.
EN

Immutable backups: stop a hacked server deleting your backups

Short answer: A backup the attacker can delete isn't much of a backup. When ransomware or an intruder gets into a server, they often use the same credentials to wipe the backups before demanding payment. Object lock, offered by major cloud storage services, makes each backup copy write-once: it can't be deleted or overwritten until a retention date you set, even by someone holding your keys. Send backups to a bucket with object lock on, give the server credentials that can upload but not change retention, set a retention period longer than you'd take to notice an attack, and test restores.

Why ordinary offsite backups aren't enough

A typical setup: a nightly job on the web server uploads a database dump and files to cloud storage. That protects you from a dead disk or a bad update. But the job needs credentials to write to the bucket, and those credentials sit on the server. An attacker who controls the server can use them to delete the bucket's contents, or overwrite backups with encrypted or empty files. Restores then fail exactly when you need them.

What you need is a copy that the server can create but not destroy.

What object lock does

Amazon S3's documentation describes Object Lock as storing objects using a write-once-read-many (WORM) model, preventing them from being deleted or overwritten for a fixed period or indefinitely. It provides two ways to keep objects locked:

  • Retention period: a date until which the object version stays locked.
  • Legal hold: a lock with no expiry date that stays until removed.

You can also set a default retention period on the bucket, so every new upload is locked automatically.

Backblaze B2's documentation describes the same idea for its storage: object lock makes data immutable until a given date, to protect against threats like ransomware, with retention from one day to 3,000 days. Many other S3-compatible services offer similar features.

Compliance mode or governance mode

AWS's docs explain the two retention modes:

  • Compliance mode: no user, including the account's root user, can overwrite or delete a locked object version, and the retention period can't be shortened. AWS notes that the only way to remove such objects before the date is to delete the whole account.
  • Governance mode: most users can't delete or change locked objects, but users with a special permission can. AWS suggests it for testing retention settings before switching to compliance mode.

For a small business, a sensible path is to start in governance mode while you get retention and costs right, then move to compliance mode, keeping the override permission away from everyday accounts.

Important detail: versions and delete markers

AWS's documentation notes that locks don't stop new versions of an object being created or delete markers being added. The locked version underneath stays intact. So an attacker can make a backup look deleted, but you can still restore the locked version. Know how to list and restore previous versions in your storage service before you need it.

Set it up step by step

  1. Create a new bucket with object lock enabled. Some services require it at creation; Backblaze allows enabling it on existing buckets too.
  2. Set a default retention period. Choose something longer than the time it might take to notice a compromise. Attackers sometimes wait before acting, so a few days is usually too short.
  3. Create separate credentials for the server that can upload new objects and read, but cannot delete objects, change retention or alter bucket settings.
  4. Keep admin credentials elsewhere, protected with two-factor authentication, never on the server.
  5. Point your backup tool at the bucket. Check that it works with object lock; tools that rewrite or "prune" old files in place may fail or need settings changes.
  6. Set lifecycle rules to delete objects after their retention ends, so storage doesn't grow forever.

Plan the cost and the retention

Locked data can't be deleted early, so you pay for its storage for the whole retention period. Big daily full backups with long retention add up. Common approaches: keep daily backups locked for a few weeks and weekly or monthly ones for longer, or use incremental backups that store only changes. Test with governance mode first, check the bill after a month, then decide.

Test the restore

An immutable backup you've never restored is still a guess. At least every few months:

  • Restore the latest backup to a spare server or a local machine.
  • Confirm the website or app starts and the data is recent.
  • Practise restoring a version "hidden" behind a delete marker.
  • Time it, so you know how long an outage would last.

What this doesn't cover

Object lock protects backup copies. You still need security updates, strong logins, two-factor authentication on hosting and cloud accounts, and monitoring. And it can't help if the data was already corrupted before it was backed up, which is another reason for keeping several restore points.

A worked example

An illustrative Phoenix dental practice's booking site and patient portal backed up nightly to cloud storage using keys stored on the server. After reading about a nearby clinic losing both site and backups, they created a new bucket with object lock in governance mode, set a 30-day default retention, and gave the server upload-only credentials. After confirming costs and rehearsing a restore, they switched to compliance mode. A test where they tried to delete backups from the server confirmed the deletions were refused.

FAQ

Can I delete a locked backup if I made a mistake?

In governance mode, with the right permission. In compliance mode, not until the retention date.

Does object lock encrypt my backups?

No. Use your storage's encryption features as well, and encrypt sensitive backups yourself.

Do I still need a second copy?

Ideally yes, in a different account or provider, so one account problem doesn't affect everything.

Will my existing backup tool work?

Many do; check its documentation for object lock or immutability support.

If your backups could be deleted by the same person who breaks into your server, tell us how they're set up and we'll help you make them immutable.

Sources

WhatsAppBook a call