The Cursor That Never Ends: Catching a Pagination Loop Before It Eats the Night

Sometimes a paginated source keeps returning a cursor that points back to a page you have already read. A naive loop reads the same pages forever and may fill a store with duplicates. This article proposes a loop witness, which is a small record that proves a cycle, plus a hard bound that guarantees the loop ends even when the witness cannot decide. It is a conditional design suggestion and not a statement about how any given API behaves.

The loop witness

During a run, keep two things.

1. A map from every cursor seen to the page index where it was first seen.
2. A fingerprint for each page, such as a hash of the ordered item keys on it, mapped to its first page index.

When a page returns a next cursor already in the map, you have a cycle witness: cursor value, first page index, current page index, and the cycle length (current minus first). Store that record with the run, since it is the evidence a person needs.

Branches

Branch A, immediate repeat: the next cursor equals the one just used. Cycle length 1. Stop at once.

Branch B, longer cycle: the next cursor equals one from several pages ago. Stop and record the length.

Branch C, cursor changes but content repeats: new cursor strings, yet the page fingerprint matches an earlier page. This may be a source that regenerates tokens. Treat one repeated fingerprint as a warning and two in a row as a stop, because short pages with coincidentally equal keys are possible in odd data.

Branch D, changing cursors and new content forever: no witness fires. This may be a legitimate huge listing or a runaway. Only the bounds decide it.

The bounds

Set two limits chosen from your own needs: a maximum page count and a maximum elapsed time. When either is reached the run stops with status bound reached, which is deliberately different from finished and from cycle detected. A bound reached run is incomplete and must not be reported as a full sync.

Worked hypothetical

A fictional source returns cursors c1, c2, c3, c2. With a page bound of 50:

• Page 1 is read with no cursor, returns c1.
• Page 2 is read with c1, returns c2. First seen c2 at index 2 (the page that returned it).
• Page 3 is read with c2, returns c3.
• Page 4 is read with c3, returns c2.

At page 4 the returned cursor c2 is already in the map, first seen at index 2. Current index is 4, so cycle length is 2. The run stops after 4 pages, far under the bound of 50, with status cycle detected and the witness stored. Items from pages 1 to 4 were applied by key, so 4 pages of reading should add no more than the distinct keys seen.

Stop conditions and follow up

On any stop, do not retry automatically with the same starting point, since the same cycle will recur. Leave the run for a person, who may try a different documented filter or contact the source owner. If the sync also writes remotely, do not repeat any write whose result is uncertain.

Proposed acceptance tests

These are proposals, not results.

1. Build a stub that returns c1, c2, c3, c2 and check the run stops on the fourth page with cycle length 2.
2. Build a stub whose cursor equals the request cursor and check cycle length 1.
3. Build a stub with fresh cursors and identical pages and check Branch C warns, then stops after the second consecutive repeat.
4. Build a stub with endless distinct pages and check the run ends at the page bound with status bound reached, not finished.
5. Build a finite stub with 5 distinct pages and check no false cycle is reported and status is finished.

A note on memory

The witness map grows with the page count, but the bound limits it. Keep cursors as stored strings, and avoid logging full tokens if they could be sensitive.

Sources