A Batch Size Ladder for a Slow Catalogue Import
When a catalogue import crawls, the tempting fix is to pick a batch size from memory or from someone else's setup. A better approach is a ladder: you start small, climb one rung at a time, and record what each rung did. Every rung is a test you can undo. This article proposes a workflow you could adapt. It does not assume your import tool offers any particular setting, and it promises no particular speedup.
Before you start: make the tests reversible
Pick a test slice of records that you can safely remove or overwrite, such as a clearly labelled set of dummy products or a copy of the catalogue. If you cannot isolate a slice, stop here. The ladder is only useful when a failed rung costs you nothing you cannot restore. Note how you would undo each trial before you run it, for example deleting the test records by their shared label.
The ladder
Choose rungs that grow in steps you can reason about. A hypothetical ladder might be 10, 25, 50, 100, 200 records per batch. Use the same test records on every rung so that differences come from the batch size, not the data. Run each rung at least twice if time allows, and note when the two runs disagree sharply, because that suggests something other than batch size is moving.
The trial log
Copy this layout into a spreadsheet or a plain text file.
Rung number Batch size Total records in test Start time and end time Elapsed minutes Records accepted Records rejected or skipped Warnings or timeouts seen (describe in words) Cleanup done (yes or no, and how) Decision: climb, hold, or stop
Fill in the row immediately after the trial, not from memory later. Elapsed time divided by records processed gives you a rough minutes-per-record figure to compare rungs. Treat it as a comparison within your own log, not as a benchmark.
Branching choices
If a rung is clearly faster per record and shows no new warnings, climb one rung. If it is only slightly faster, hold and repeat once. Small gains can be noise. If it is slower or shows new warnings, step back to the previous rung and mark that rung as your candidate. If rejections appear that did not appear before, stop climbing and inspect those records first, since the cause may be the data and not the size.
The stop rule
Write this before the first trial and do not renegotiate it mid-run. A hypothetical version: stop climbing when any of these happens. Per-record time gets worse for two rungs in a row. Any timeout or error appears that the previous rung did not show. Rejected records increase. You reach your last planned rung. Also stop immediately if cleanup of a trial fails, and fix that before anything else.
A worked example
Imagine a catalogue of several thousand products, with a test slice of 200 labelled dummy records. Rung 1, batch size 10, takes 14 minutes with no warnings. Rung 2, batch size 25, takes 9 minutes, clean. Rung 3, batch size 50, takes 7 minutes, clean. Rung 4, batch size 100, takes 7 minutes with one timeout warning. Rung 5 is never run, because the stop rule triggered on the new warning. The candidate is 50. These numbers are invented for illustration only. Your own log will say something different.
Acceptance checks
Before adopting the candidate size for the real import, confirm the following. The log has a complete row for every trial. The candidate rung ran cleanly at least twice. All test records were cleaned up or are clearly labelled. The accepted count matches the number you expected. You have written down the next action if the real import shows warnings, such as dropping back one rung.
A caution on generalising
A size that works on a test slice may behave differently on the full catalogue, with larger images, longer descriptions or busier hours. Consider starting the real import on the candidate size with the first batch watched closely, and keep the previous rung as your fallback. Treat the result as a working setting, revisited whenever the catalogue or the import path changes.