Snapshot is a feature which can save the current state of integration step for the future reference. This feature can be used to prevent the repetitive execution of the same integration state. It is a JSON object which can contain any information deemed useful in preserving the current state.

Please note: Snapshot has a hard limit of 5Kb in size to prevent its misuse as an intermediate database. If the snapshot exceeds 5Kb limit your integration task will not be scheduled and an Error will be reported: Oops Failed to schedule execution of task "Name of your task".

Please use an external database like MongoDB, Redis, MySQL, etc. instead, if this limitation is affecting your workflow.

What can snapshot do for your integration?

Snapshot feature can help you to:

  • Avoid the data duplication when the same data is not synced several times which can potentially cause data loss by overwriting records at some stage.
  • Save on processing time and costs in integration by transferring only the newest records instead of the whole dataset again.

Here is a scenario which explains the best one of the common usages of the snapshot feature.

Let's just consider a very common situation: Two systems need to be synced and every change in the first system should be reflected in the second one.

To keep both systems in instantaneous sync would not be almost unreachable but also be resource intensive. This is the reason both systems should connect in some time intervals. System one or the source will be POLLED for a new information at some specific intervals (usually every 3 minutes for ordinary flows) in order to transfer it to the System two. The question is how would our platform know when was the last sync and what was transferred from System one to System two during that last sync?

It is doubtful that somebody would be constantly making notes of the changes in both systems, but it is a thought in the right direction. On our platform, it is possible to record the last state of your integration flow in the snapshot and read it before requesting for an update. Think about the snapshot as a piece of paper where only your last action is recorded - nothing more.

Snapshot usage example: Querying objects by timestamp

This is an algorithm to query objects from systems:

Query all objects since 1970, 1 Jan. This is done to establish the first point of reference. Usually, this is the stage of first global sync between two systems.
Iterate over all objects. In some cases this stage can take a while due to the amount of data in the source system, therefore it is recommended to use paging if the originating API supports it.
Calculate max update date based on returned objects. You must not calculate the max date as current time as you can miss some objects. This stage is also dependent on the originating API properties.
Emit the new max date as snapshot after iterating through all objects.

This is the general algorithm, but how exactly we could achieve this? One distinct possibility is to first check for the snapshot existence. If it exists, write the first sync date to be 1970, 1st of January like this:

params.snapshot = snapshot;

if (!params.snapshot || typeof params.snapshot !== "string" || params.snapshot === '') {
  // Empty snapshot
      console.warn("Warning: expected string in snapshot but found : ", params.snapshot);
      params.snapshot = new Date("1970-01-01T00:00:00.000Z").toISOString();
 var newSnapshot = new Date().toISOString();


Again one important fact would need to be considered that every time snapshot file needs to be overwritten and it should not be used as a temporary database of values since you are risking of breaching the upper limit for the snapshot (5KB) in no time.

Next: How to create a Snapshot