When the Archive Is Too Big: A Manifest That Lets You Offer Parts Honestly

Sometimes a requested archive cannot be generated as one file because it exceeds a configured limit. Failing outright wastes the work. Offering a few parts is useful only if the visitor can tell whether they hold everything. This article proposes a manifest that makes completeness checkable. It is a conditional design, to be adapted to your own limits and storage.

The primary artifact: the manifest

Generate a manifest before any part is built. It contains:

Export identifier and the time the file list was fixed.
Expected total file count and total bytes.
Part limit in bytes, as configured.
For each part: part number, ordered file list, file count and byte total.
Part status: planned, built, failed.

The manifest is the single source for the page that lists downloads. A part is shown as downloadable only when its status is built.

Decision branches

Exactly one applies to each export.

Everything fits under the limit: build one archive. No manifest page is needed.

Total exceeds the limit and every file is individually under it: pack files in their existing order into parts, never splitting a single file.

One file is over the limit by itself: do not cut it silently. Offer that file as a separate direct download if your storage allows it, or leave it out and say so in the manifest as an excluded item. The export is then not complete, and the page must say so.

Worked hypothetical example

The limit is 100 MB per part. Nine files, in order: A 40, B 35, C 30, D 60, E 20, F 45, G 10, H 25, I 90 MB. Total 355 MB.

Packing in order without splitting files:

Part 1: A + B = 75 MB (2 files). Adding C would make 105, so it moves on. Part 2: C + D = 90 MB (2 files). Part 3: E + F + G + H = 100 MB (4 files). Exactly at the limit counts as allowed. Part 4: I = 90 MB (1 file).

Check: 2 + 2 + 4 + 1 = 9 files, and 75 + 90 + 100 + 90 = 355 MB. Both match the manifest's expected totals, so the plan is consistent.

Suppose part 3 fails to build. The page shows parts 1, 2 and 4 as available, part 3 as failed, and the banner reads "Incomplete: 5 of 9 files available" (2 + 2 + 1). It must not say "download complete".

Completeness check

After all builds, compute the check from the built parts, not from the plan: the number of distinct files across built parts, and their byte sum. Compare against the manifest's expected values. Show "Complete" only when both match and no item is excluded. Include the manifest in the visitor's download, for example as a plain text file listing each part and its files, so they can verify later. If you use checksums, treat them as a way to detect damaged downloads, not as proof of who produced the file.

Stop conditions

If the file list changes while parts are building, stop and rebuild the manifest. Do not mix old and new parts.
If a part fails, do not automatically retry in a way that could duplicate externally visible effects such as sending a link by email. A rebuild of a local part is safe to retry a bounded number of times. Beyond that, mark it failed.
If total file count in the built parts exceeds the expected count, flag a defect and withhold "Complete".

Proposed acceptance tests (not yet performed)

Run the nine file example and check the four parts hold 2, 2, 4 and 1 files.
Force part 3 to fail and check the banner says 5 of 9 and never "Complete".
Add a 120 MB file and check it appears as an excluded item or separate download, not inside a cut part.
Remove a built part from the list and check the completeness check turns negative.
Modify the file list mid-build and check the rebuild is triggered.

Limits

The manifest proves your own parts add up to your own plan. It cannot show the visitor's extraction tool handled them correctly. Keep the wording modest: "All 4 parts are available and match the list."