Five kinds of API pagination, and how to configure each

Every REST API paginates and no two do it the same way. One increments page. One wants startAt. One hands you an opaque cursor. One returns a complete URL for the next call. One expects a time window and gives you nothing else.

That used to mean a preprocessor script per API, which is code somebody has to own two years later when the API changes. The HTTP connector handles all five patterns as connection settings.

Pick the type

page increments a page number in the query string. Set Page Parameter Name (page) and optionally Page Size Parameter Name (pageSize).

offset increments an offset. Set Offset Parameter Name (offset, start, from).

cursor pulls a token out of each response and sends it with the next request. Set Cursor JSON Path to find the token and Cursor Parameter Name to send it back (after, nextPageToken).

nextLink follows a complete URL the API hands you. Set Next Link JSON Path.

time walks a date range. Set Start Time, an optional End Time, Time Parameter Name, and Time Format (Java) using DateTimeFormatter syntax.

Settings that apply to all of them

Data JSON Path points at the array of records, so nothing downstream has to unwrap an envelope. Max Pages caps the run and is worth setting while you are still testing. Start Page or Offset and Page Size do what they say. Stop Condition JSON Path ends the loop when a value in the body signals there is nothing left.

Microsoft Graph, in four settings

Graph returns @odata.nextLink, so the whole configuration is:

URL:                  https://graph.microsoft.com/v1.0/users
Pagination Type:      nextLink
Data JSON Path:       /value
Next Link JSON Path:  /@odata.nextLink

The connector follows the link until the API stops sending one.

A page-based example

The Rick and Morty API reports its page count in the body, and the connector can read that to know where to stop:

URL:                         https://rickandmortyapi.com/api/character
Pagination Type:             page
Data JSON Path:              /results
Page Parameter Name:         page
Max Pages Path in Response:  /info/pages

The awkward ones

Some APIs want pagination keywords inside the query itself rather than as URL parameters. QuickBooks is the usual example, with STARTPOSITION and MAXRESULTS embedded in a SQL-like query string. Turn on Inject offset and page size into query string and the connector puts them where the API expects. It applies to the offset and page types.

Then there is the API that returns something which is not quite a URL: a partial path, a link expression, structured metadata you have to assemble yourself. That is what the Pagination URL Preprocessor is for. The script receives two variables, url with what the connector computed and nextUrl with the raw value from the response. Return a string and it becomes the next request URL.

That is the one place scripting survives, and it is deliberately small. You correct the URL and hand it back.

Where 8.8.1 fits

Automatic pagination has been in the connector for a while. Version 8.8.1, released in January, is where it got sharp: better nextLink handling and the derived-URL edge cases above, which were exactly the cases that used to push people back into writing a script.

The full reference, with worked examples for Jira, HubSpot, and Mixpanel, is in Work with Paginated APIs Using the HTTP Connector.