Lorixen.
EN

S3 lifecycle rules: expire or archive old files automatically

Short answer: S3 lifecycle rules tell the storage service to move or delete objects automatically after a set time, so old uploads, exports, logs and backups don't pile up and quietly grow your bill. Use expiration actions to delete things you no longer need, transition actions to move rarely used files to cheaper storage classes, and a rule to clean up incomplete multipart uploads. Scope each rule with a prefix or tag, and test on a copy before applying it to real data.

Why storage bills creep up

Object storage is cheap per gigabyte, which is exactly why nobody watches it. Over a couple of years a typical business app accumulates:

  • Temporary exports and reports users downloaded once.
  • Original-size photos alongside the resized versions actually used.
  • Daily database backups, all of them, forever.
  • Log files shipped from servers.
  • Fragments of uploads that never finished.

None of that is free, and some of it carries risk too: old exports may contain customer data you'd rather not keep. Lifecycle rules are the built-in way to clean up without writing and running your own scripts.

The two kinds of action

Amazon's S3 documentation describes a lifecycle configuration as a set of rules defining actions applied to a group of objects, with two types of action:

  • Transition actions move objects to another storage class after a set time, for example moving files to an infrequent-access class 30 days after creation.
  • Expiration actions define when objects expire, and S3 deletes expired objects on your behalf.

Other object storage services offer similar lifecycle features. The details and names differ, so check your provider's documentation, but the planning below applies to all of them.

Start by sorting your data

Before writing any rules, list what's in the bucket and how long each kind needs to live. A simple table helps:

Data Who needs it Keep for Action
Temporary exports The user, once A few days Expire
Original photo uploads Rarely, for re-processing Long term Transition to a cheaper class
Resized images Website visitors While in use Keep in standard storage
Database backups You, in an emergency A set number of weeks Transition, then expire
Server logs Debugging A few weeks or months Expire

The retention periods are your business decisions. Agree them with whoever is responsible for the data, then encode them as rules.

Scope rules carefully

A lifecycle rule applies to a filter, usually a key prefix such as exports/ or an object tag. This is where mistakes happen. A rule meant for tmp/ that's accidentally set to apply to the whole bucket will delete everything older than its age limit.

Good habits:

  • Keep different data under different prefixes from the start, such as uploads/originals/, uploads/resized/, exports/ and backups/.
  • Use tags for exceptions, for example tagging files a customer asked you to keep.
  • Name rules clearly, such as "expire-exports-7-days", so the intent is obvious later.
  • Apply new rules to a test bucket first, with copies of real key names.

Lifecycle actions don't happen at the exact second an object reaches its age; the provider processes them in the background. Don't rely on them for anything time-critical.

Clean up incomplete multipart uploads

Large files are often uploaded in parts. If an upload is abandoned, the parts already uploaded can stay in the bucket, using storage, even though no complete object appears in normal listings. Amazon's documentation recommends a lifecycle rule using the AbortIncompleteMultipartUpload action to minimise storage costs, with a set number of days after which unfinished uploads are cleaned up.

This is one rule almost every bucket that accepts large uploads should have. It's easy to forget because the leftover parts are invisible in normal file listings.

Watch the small print on cheaper classes

Moving data to cheaper storage isn't always cheaper overall. Things to check in your provider's current documentation and pricing:

  • Small objects. Amazon's documentation says objects smaller than 128 KB won't transition to another storage class by default, and it doesn't recommend transitioning them, because each transition is charged as a request and the saving is tiny.
  • Minimum storage periods. Some classes charge for a minimum number of days even if you delete the object sooner.
  • Retrieval costs and delays. Archive classes can take time to restore files and may charge per retrieval.

For files a website shows to visitors, keep them in standard storage. Transition only what's genuinely rarely read.

Versioned buckets need extra rules

If versioning is switched on, deleting or overwriting an object keeps the old version as a noncurrent version. Lifecycle rules can expire noncurrent versions after a set number of days, separately from current ones. Without that rule, a versioned bucket can grow indefinitely even though the "current" files look small. Keep versioning for important data, but give old versions an end date.

Check that rules are working

After a rule has been in place for a while:

  1. Compare total bucket size and object count over time.
  2. Use your provider's storage analytics or inventory reports to see what's in each prefix.
  3. Spot-check that files under protected prefixes are still there.
  4. Review rules whenever you add a new type of data.

A worked example

Take a hypothetical Perth equipment-hire app that stored signed hire agreements, customer ID photos, PDF exports and nightly database backups in one bucket under a flat structure. The bill had grown steadily for two years. The team reorganised new files under prefixes, added rules to expire exports after seven days and backups after a period the owner chose, transition originals to an infrequent-access class, abort incomplete uploads after a few days and expire noncurrent versions. Growth flattened, and old exports with customer details stopped accumulating.

FAQ

Do lifecycle rules apply to existing files?

Yes. Rules apply to matching objects based on their age, including ones created before the rule. That's why scoping must be right before you switch a rule on.

Can I undo an expiration?

Not in an unversioned bucket. Once deleted, it's gone. Test on a copy first.

Are lifecycle rules free?

Configuring them generally is, but transitions can incur request charges. Check your provider's current pricing.

Should backups expire automatically?

Usually yes, with a retention period you've chosen deliberately, so storage stays bounded.

If your storage bill keeps growing and nobody's sure what's in the bucket, ask us to review your storage and set up lifecycle rules.

Sources

WhatsAppBook a call