Before we can proceed any further one should clearly examine the API provider's documentation to get familiarised with the accepted method of pagination usage.
In order to avoid producing too many data, a component developer should use paging which is done in a combination with the snapshot feature.
Implementing paging with the help of snapshot
First, we would need to setup the snapshot object which would only have 2 properties: next
timestamp. The property
timestamp is used to record the most recent update time of an object in the target system and the property
nextPage is used to remember the index of the next page to retrieve. Here is the algorithm behind:
- At the beginning snapshot is empty, so that trigger queries based on timestamp
- If the API supports paging, trigger checks if the result is paged. If yes then the next page to retrieve is calculated and saved into the
nextPageproperty in the snapshot.
- On next iteration, trigger checks if
nextPageis available in the snapshot. If so it queries now the next page and overwrites the
nextPagewith a new value. At the same time, it should preserve the
timestampfor the later comparison.
- After iterating through all the pages, the
nextPageparameter is removed from the snapshot. However, the
timestampproperty must stay in the snapshot.
- It is important to define max
timestampthrough all the pages. This is accomplished either by filtering or by comparing all the
timestampvalues in all pages with the
timestampin the snapshot.