How to Prevent Duplicate Processing in n8n Workflows

You've built a workflow that processes incoming orders. It works great, until it processes the same order twice. The customer gets double-charged, your inventory count is wrong, and your downstream system has duplicate records.

Sound familiar? Duplicate processing is the most common reliability problem in production n8n workflows, and it affects everyone from solo automation builders to enterprise teams.

In this guide, we'll look at why duplicates happen, how people typically work around them, and how to solve it properly with persistent deduplication.

Why Duplicates Happen in n8n

Duplicates don't just come from bugs. They come from the nature of distributed systems:

• Webhooks fire twice. Payment providers like Stripe explicitly state they may send the same event multiple times. Shopify webhooks can retry on timeout.
• Polling overlaps. A Schedule Trigger fires every 5 minutes. If the previous execution is still running, both executions fetch and process the same batch of records.
• Retries after failures. Your workflow fails halfway through processing 50 items. On retry, it starts from the top and re-processes the 30 items that already succeeded.
• Manual re-runs. Someone clicks "Execute Workflow" to test, not realizing it will re-process live data.

n8n's built-in Remove Duplicates node helps within a single execution, it can deduplicate items in the current batch. But it has no memory across executions. If the same order came in yesterday and comes in again today, the Remove Duplicates node treats it as new.

The DIY Approaches (and Why They Break)

Approach 1: Check the Destination System

The most common workaround: before creating a record, check if it already exists.

Webhook → HTTP Request (check if order exists) → IF → Create Order

Problems:

• Race condition: two executions check at the same time, both see "doesn't exist," both create the record
• Adds latency to every execution (even for genuinely new items)
• Different APIs have different lookup semantics, not always reliable
• Doesn't work when the destination doesn't support idempotency

Approach 2: Google Sheets / Airtable as a Ledger

Store processed IDs in a spreadsheet:

Webhook → Google Sheets (lookup ID) → IF → Process → Google Sheets (record ID)

Problems:

• Google Sheets API has rate limits (60 requests/minute on the free tier)
• No atomicity, the check and the record happen in separate steps. Crash between them = gap
• Grows unbounded with no built-in cleanup
• Not designed for this use case, slow, fragile, expensive at scale

Approach 3: n8n Variables / Static Data

Use workflow static data or n8n variables to track processed items:

// In a Code node
const processed = $getWorkflowStaticData('global');
if (processed[itemId]) return []; // skip
processed[itemId] = true;
return items;

Problems:

• Static data is per-workflow, if multiple workflows process the same data, they can't share state
• Lost on workflow reimport or n8n upgrade in some configurations
• No built-in expiry or cleanup
• Not designed for high-volume tracking

The Better Way: Persistent Deduplication with 8kit

8kit's Uniqs pattern gives you persistent, cross-execution deduplication that was purpose-built for this exact problem.

Here's how it works:

Step 1: Create a Uniq Collection

A Uniq Collection is a named set that tracks which values you've already seen. Create one for your use case, for example, processed-orders.

In n8n, drag in the 8kit node and select:

• Resource: Uniq Collection
• Operation: Create Uniq Collection
• Name: processed-orders

You only need to do this once. The collection persists across all workflow executions.

Step 2: Check Before Processing

Before you process an item, check if its unique identifier is already in the collection:

• Resource: Uniq
• Operation: Check Uniq Values
• Collection: processed-orders
• Value: {{ $json.orderId }}

The 8kit node routes items to two outputs:

• Output 1 (Existing): The value was already in the collection, this is a duplicate
• Output 2 (Non-existing): The value is new, safe to process

Step 3: Mark as Processed

After successful processing, add the value to the collection:

• Resource: Uniq
• Operation: Add to Uniq
• Collection: processed-orders
• Value: {{ $json.orderId }}

The Complete Workflow

Webhook → Check Uniq Values → [Existing] → Skip (log duplicate)
 → [Non-existing] → Process Order → Add to Uniq → Done

That's it. Three nodes added to your workflow, and you have persistent deduplication that:

• Works across executions, workflow restarts, and n8n upgrades
• Handles concurrent executions without race conditions
• Works across multiple workflows (share the same collection)
• Provides a dashboard to inspect what's been tracked
• Has no rate limits or scaling concerns

Real-World Example: Shopify Order Processing

Here's a concrete example. You're syncing Shopify orders to your ERP:

Before 8kit:

Shopify Trigger → HTTP Request (check ERP for order) → IF exists → Create in ERP

Problem: Shopify webhooks can fire twice during high-traffic events (Black Friday, flash sales). Your ERP check has a 200ms round-trip, creating a race window.

After 8kit:

Shopify Trigger → 8kit Check Uniq (order ID) → [New] → Create in ERP → 8kit Add to Uniq
 → [Exists] → Log & Skip

The dedup check happens in the 8kit server, which handles concurrent requests atomically. No race window, no false positives.

When to Use Each Approach

Scenario	Recommendation
Deduplicating items within a single batch	n8n's Remove Duplicates node (built-in)
Preventing re-processing across executions	8kit Uniqs
Sharing dedup state across workflows	8kit Uniqs (shared collections)
Idempotent webhook handling	8kit Uniqs with webhook event ID as the value
Low-volume, non-critical workflows	Destination system check may be acceptable

Try It Yourself

1. Install the 8kit node: Settings > Community Nodes > Install n8n-nodes-8kit
2. Import the demo workflow: Grab demo/01-deduplication.json from the GitHub repo
3. Connect to 8kit: Point to your 8kit server (or try the live demo)

The demo workflow shows email deduplication end-to-end, run it twice and see the second execution get correctly routed to the "skip" path.

What's Next

Deduplication is one of four reliability patterns 8kit provides. If you're also dealing with:

• Race conditions between concurrent executions, see Solving Race Conditions in n8n (coming soon)
• Cross-system ID mapping for Shopify/ERP/CRM syncs, see Mapping IDs Across Systems in n8n (coming soon)
• Re-processing entire datasets on every poll, see Incremental Sync in n8n (coming soon)

Read The 4 Patterns Every Production n8n Workflow Needs for the full picture.

---

8kit is a free, open-source toolkit for building reliable n8n workflows. Get started at 8kit.io.