One Stored File, Many Owners: Deduplicating Uploads Without Sharing Access

Storing the same bytes twice wastes space. Merging two customers' uploads into one record is worse, because it can merge who may see them. This article proposes a design that separates what a file is from who may use it. These are conditional recommendations to adapt, not a description of any shipped system.

The artifact: content identity versus access ownership

Keep two kinds of record.

Content blob. Identity is a digest of the bytes plus the byte length. It has a verified flag and a reference count. It has no owner, no name and no permissions.
Asset record. It holds the workspace, the uploader, the display name, the permission list, the created time, a deleted flag, and a pointer to one blob.

Rule: permission checks read only the asset record. A blob is never addressable by clients. A download is always requested through an asset id, and the check happens on that asset.

Decision branches

When an upload finishes, compute the digest on the server side from the stored bytes. Do not trust a digest the browser reports. Then pick exactly one branch:

A. No blob has this digest and length. Create a blob, then a new asset record.
B. A verified blob matches, and no live asset exists for this workspace with the same name and uploader. Create a new asset record pointing at the existing blob. Discard the newly uploaded copy only after the record is saved.
C. A verified blob matches, and the same uploader already has a live asset in this workspace with the same name. Return the existing asset and tell the user it already exists. Create nothing.
D. The digest matches but the length differs, or the matching blob is not yet verified. Keep the new bytes as a separate blob and flag the pair for review. Never merge on a digest alone.

Worked hypothetical example

Five uploads arrive:

U1: Ana, workspace Red, logo.png.
U2: Ben, workspace Blue, identical bytes to U1.
U3: Ana, workspace Red, the same bytes again, named logo.png.
U4: Cai, workspace Red, identical bytes named logo-old.png.
U5: Ana, workspace Red, logo.png with one byte changed.

Outcomes: U1 takes branch A, so blob H1 and asset 1 are created. U2 takes branch B, giving asset 2 on H1. U3 takes branch C, so nothing new is created and asset 1 is returned. U4 takes branch B, giving asset 3 on H1. U5 has a new digest, so branch A creates blob H2 and asset 4.

Result: 5 uploads, 4 asset records, 2 blobs. H1 has reference count 3 and H2 has 1. If Ana deletes asset 1, only that record is marked deleted. H1 keeps count 2 and Ben and Cai still download normally. Blue never sees anything about Red.

Stop conditions

Stop and route to review instead of continuing if any of these hold:

The server cannot compute a digest from the stored bytes.
A reference count would go below zero.
A blob is found with zero live references and a recent reference write whose outcome is unknown. Do not delete it automatically, and do not repeat the uncertain write blindly. Re-read the current state first.

Layered constraints, not guarantees

Exposure is reduced by several constraints together: permission checks on asset records only, no client-visible blob addresses, no cross-workspace hints in messages, and audit entries naming the asset rather than the blob. Any single layer can have a bug, so test them separately.

Proposed acceptance tests (to be run, none performed here)

Replay the five uploads above and expect 4 assets, 2 blobs and counts 3 and 1.
Delete asset 1 and expect assets 2 and 3 to download, and H1 to remain with count 2.
As Ben, request asset 1 by id and expect refusal that reveals nothing about its existence.
Upload a file whose digest matches but whose declared length differs, and expect branch D with no merge.
Delete the last reference to H2 and expect the blob to be removed only after a later re-check finds count 0.
Interrupt a branch B save halfway and expect no orphan reference and no duplicate asset on retry after state is re-read.

If any test fails, the model is not ready: fix the record separation before saving any more storage.