Incremental Sync in n8n: Stop Re-Processing Your Entire Dataset

Your n8n workflow syncs customers from Shopify to your CRM every 15 minutes. Each run fetches all 8,000 customers and pushes them to HubSpot. But only 3 changed since the last run.

You're burning 7,997 unnecessary API calls every 15 minutes. That's 767,712 wasted calls per day. Your Shopify rate limit is crying.

This is the incremental sync problem, and it's one of the most overlooked performance killers in n8n automations.

The Full-Scan Anti-Pattern

Most n8n sync workflows follow this pattern:

Schedule Trigger (every 15 min)
→ Fetch ALL records from Source
→ Loop through each record
→ Update or Create in Target

It works. It's simple. And it falls apart as your data grows.

Why full scans hurt

API rate limits. Shopify gives you 40 requests per second (REST) or 1,000 points per second (GraphQL). Fetching 8,000 customers at 250 per page = 32 requests just for the read side. Do that every 15 minutes while other workflows also need Shopify access, and you're rate-limited by lunchtime.

Processing time. Pushing 8,000 records to HubSpot takes time, even with batching. Your 15-minute schedule overlaps with the previous run. Now you have concurrency issues on top of performance issues.

Unnecessary target system load. Your CRM, ERP, or database receives 8,000 update calls when only 3 records changed. Many APIs count these against your plan limits regardless of whether the data actually changed.

Cost. If you're on usage-based pricing for any API in the chain, you're paying for 99.96% wasted work.

DIY Incremental Sync Approaches

Approach 1: Filter by updated_at with a hardcoded window

Schedule Trigger (every 15 min)
→ Shopify: Get customers updated in last 20 minutes
→ Process only those customers

The overlap window (20 min for a 15-min schedule) catches items that might have been missed. But:

• You're still re-processing items from the overlap window
• If the workflow is down for an hour, you miss everything outside the 20-minute window
• If the schedule interval changes, you need to update the filter window manually

Approach 2: n8n Static Data timestamp

// In a Function node
const staticData = $getWorkflowStaticData('global');
const lastRun = staticData.lastRunTimestamp || '2024-01-01T00:00:00Z';
staticData.lastRunTimestamp = new Date().toISOString();
return [{ json: { since: lastRun } }];

Better. But:

• Updates before processing: The timestamp is saved at the start of the run. If the workflow crashes midway, items processed after the saved timestamp but before the crash are never retried.
• Per-workflow: If you have three workflows that all need to know "last sync time for Shopify customers," you're maintaining three separate timestamps.
• Lost on reimport: Export and reimport the workflow, and lastRunTimestamp resets. Your next run does a full scan.
• No visibility: You can't inspect static data without a dedicated Function node.

Approach 3: External database timestamp

Store the last-sync timestamp in your own database:

Schedule Trigger
→ Read last_sync from DB
→ Fetch records updated since last_sync
→ Process records
→ Write new last_sync to DB

This is actually solid, but you need database access from n8n, schema setup, and you're managing infrastructure for what should be a simple "remember when I last ran" operation.

The 8kit Approach: Temporal (Last Updated)

8kit's Temporal pattern gives you persistent, named timestamps that track when each resource or sync job was last processed. No database, no static data, no hardcoded windows.

How it works

Schedule Trigger
→ 8kit Temporal: Get "shopify-customers-sync" timestamp
(returns: 2026-04-08T12:45:00Z)
→ Shopify: Get customers updated since 2026-04-08T12:45:00Z
→ Process changed customers
→ 8kit Temporal: Set "shopify-customers-sync" to now

The timestamp is:

• Persistent, survives workflow restarts, reimports, and server reboots
• Named, shared across any workflow that needs it
• Visible, inspect and manage via the 8kit dashboard
• Atomic, set it only after successful processing

Key principle: update timestamp LAST

This is critical. Only update the Temporal timestamp after all processing succeeds:

Schedule Trigger
→ 8kit Temporal: Get last sync time
→ Fetch changed records
→ Process all records
→ IF all succeeded:
→ 8kit Temporal: Update timestamp
ELSE:
→ Error handling (the timestamp stays unchanged,
so the next run will re-fetch these records)

If the workflow crashes after fetching but before updating the timestamp, the next run automatically re-fetches the same records. No data loss. No manual intervention.

Per-resource timestamps

For workflows that sync individual resources (not batch jobs), use per-resource timestamps:

Webhook (Shopify customer update)
→ 8kit Temporal: Get "customer:{{ $json.id }}:last-sync"
→ IF $json.updated_at > last-sync:
→ Process the update
→ 8kit Temporal: Set "customer:{{ $json.id }}:last-sync"
ELSE:
→ Skip (already synced a more recent version)

This handles out-of-order webhook delivery, if you receive an older version of a customer record after already processing a newer one, the timestamp comparison catches it.

Combining with Other 8kit Patterns

Incremental sync works best when combined with dedup and locking:

Schedule Trigger
→ 8kit Temporal: Get last sync time
→ Fetch changed records from source
→ Split Into Batches
→ 8kit Uniqs: Skip if already processed this exact version
→ 8kit Exclusivity: Lock this resource
→ 8kit Lookup: Resolve target system ID
→ Create or Update in target
→ 8kit Lookup: Store mapping (if new)
→ 8kit Exclusivity: Release lock
→ 8kit Temporal: Update sync timestamp

This gives you:

• Only changed records fetched (Temporal)
• No double-processing within the batch (Uniqs)
• No race conditions with concurrent runs (Exclusivity)
• Instant cross-system ID resolution (Lookups)

Performance Impact

Real-world improvement for a typical Shopify → ERP sync:

Metric	Full scan	Incremental (8kit)
Records fetched per run	8,000	~10-50
Source API calls	32+	1-2
Target API calls	8,000	10-50
Execution time	12-15 min	15-30 sec
Daily API calls	768,000+	~5,000

That's a 99%+ reduction in unnecessary API calls.

Migration Path

You don't need to rewrite your workflows. The migration from full-scan to incremental is three steps:

• Add a Temporal GET node before your data fetch, and use the returned timestamp as a filter
• Add a Temporal SET node after your processing completes
• First run: The Temporal node returns a default timestamp (epoch or your chosen start date), so the first run does a full scan, then every subsequent run is incremental

Your existing processing logic stays exactly the same. You're just changing which records enter the pipeline.

Getting Started

Pick your highest-volume sync workflow, the one that fetches the most records on each run. Add two 8kit Temporal nodes (get before fetch, set after processing). Run it once to establish the baseline timestamp. Watch your execution times drop.

---

8kit provides four enterprise automation patterns for n8n: deduplication, cross-system mapping, distributed locking, and change tracking. Learn more at 8kit.io